Nov 03 01:50:29.607037 ip-10-0-139-26 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 01:50:29.966749 ip-10-0-139-26 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 01:50:29.966749 ip-10-0-139-26 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 01:50:29.966749 ip-10-0-139-26 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 01:50:29.966749 ip-10-0-139-26 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 01:50:29.970849 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970624    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:50:29.970849 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970848    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:50:29.970936 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970854    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:50:29.970936 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970858    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:50:29.970936 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970861    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:50:29.970936 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970863    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:50:29.970936 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970866    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:50:29.970936 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970869    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:50:29.970936 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970872    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:50:29.970936 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970874    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:50:29.970936 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970877    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:50:29.970936 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970879    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:50:29.970936 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970882    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:50:29.970936 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970885    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:50:29.970936 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970887    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:50:29.970936 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970890    2571 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:50:29.970936 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970892    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:50:29.970936 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970895    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:50:29.970936 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970897    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:50:29.970936 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970902    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:50:29.970936 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970904    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:50:29.970936 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970907    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:50:29.971375 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970909    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:50:29.971375 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970912    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:50:29.971375 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970915    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:50:29.971375 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970917    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:50:29.971375 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970933    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:50:29.971375 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970936    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:50:29.971375 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970939    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:50:29.971375 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970942    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:50:29.971375 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970944    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:50:29.971375 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970947    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:50:29.971375 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970950    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:50:29.971375 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970952    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:50:29.971375 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970955    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:50:29.971375 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970958    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:50:29.971375 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970960    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:50:29.971375 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970963    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:50:29.971375 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970965    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:50:29.971375 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970968    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:50:29.971375 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970971    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:50:29.971858 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970973    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:50:29.971858 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970975    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:50:29.971858 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970979    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:50:29.971858 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970983    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:50:29.971858 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970986    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:50:29.971858 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970989    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:50:29.971858 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970991    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:50:29.971858 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970993    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:50:29.971858 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970996    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:50:29.971858 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.970998    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:50:29.971858 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971001    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:50:29.971858 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971004    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:50:29.971858 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971006    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:50:29.971858 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971008    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:50:29.971858 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971012    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:50:29.971858 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971015    2571 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:50:29.971858 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971017    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:50:29.971858 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971020    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:50:29.971858 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971022    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:50:29.971858 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971026    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:50:29.972323 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971028    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:50:29.972323 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971030    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:50:29.972323 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971034    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:50:29.972323 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971037    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:50:29.972323 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971040    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:50:29.972323 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971042    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:50:29.972323 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971045    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:50:29.972323 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971048    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:50:29.972323 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971051    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:50:29.972323 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971054    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:50:29.972323 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971056    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:50:29.972323 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971059    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:50:29.972323 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971061    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:50:29.972323 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971064    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:50:29.972323 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971066    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:50:29.972323 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971069    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:50:29.972323 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971071    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:50:29.972323 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971073    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:50:29.972323 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971076    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:50:29.972808 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971078    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:50:29.972808 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971081    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:50:29.972808 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971083    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:50:29.972808 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971086    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:50:29.972808 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971088    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:50:29.972808 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971091    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:50:29.972808 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971094    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:50:29.972808 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971097    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:50:29.972808 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971100    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:50:29.972808 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971104    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:50:29.972808 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971107    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:50:29.972808 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971112    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:50:29.972808 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971114    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:50:29.972808 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971117    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:50:29.972808 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971119    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:50:29.972808 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971122    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:50:29.972808 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971124    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:50:29.972808 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971126    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:50:29.972808 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971129    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:50:29.973239 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971132    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:50:29.973239 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971135    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:50:29.973239 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971138    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:50:29.973239 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971142    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:50:29.973239 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971146    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:50:29.973239 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971149    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:50:29.973239 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.971152    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:50:29.973239 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972749    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:50:29.973239 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972757    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:50:29.973239 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972761    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:50:29.973239 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972765    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:50:29.973239 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972768    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:50:29.973239 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972771    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:50:29.973239 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972773    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:50:29.973239 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972776    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:50:29.973239 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972779    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:50:29.973239 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972781    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:50:29.973239 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972784    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:50:29.973651 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972786    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:50:29.973651 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972789    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:50:29.973651 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972793    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:50:29.973651 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972796    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:50:29.973651 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972799    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:50:29.973651 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972802    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:50:29.973651 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972804    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:50:29.973651 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972807    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:50:29.973651 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972810    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:50:29.973651 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972812    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:50:29.973651 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972815    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:50:29.973651 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972817    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:50:29.973651 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972820    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:50:29.973651 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972823    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:50:29.973651 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972826    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:50:29.973651 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972830    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:50:29.973651 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972834    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:50:29.973651 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972837    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:50:29.973651 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972841    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:50:29.973651 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972844    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:50:29.974161 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972847    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:50:29.974161 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972850    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:50:29.974161 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972852    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:50:29.974161 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972855    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:50:29.974161 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972857    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:50:29.974161 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972860    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:50:29.974161 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972862    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:50:29.974161 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972865    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:50:29.974161 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972868    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:50:29.974161 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972870    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:50:29.974161 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972873    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:50:29.974161 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972875    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:50:29.974161 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972878    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:50:29.974161 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972881    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:50:29.974161 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972886    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:50:29.974161 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972889    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:50:29.974161 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972891    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:50:29.974161 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972894    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:50:29.974161 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972896    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:50:29.974161 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972899    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:50:29.974628 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972902    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:50:29.974628 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972905    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:50:29.974628 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972907    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:50:29.974628 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972910    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:50:29.974628 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972912    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:50:29.974628 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972915    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:50:29.974628 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972917    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:50:29.974628 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972920    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:50:29.974628 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972922    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:50:29.974628 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972924    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:50:29.974628 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972927    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:50:29.974628 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972929    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:50:29.974628 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972932    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:50:29.974628 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972934    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:50:29.974628 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972937    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:50:29.974628 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972939    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:50:29.974628 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972941    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:50:29.974628 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972944    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:50:29.974628 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972946    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:50:29.975095 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972949    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:50:29.975095 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972952    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:50:29.975095 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972954    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:50:29.975095 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972957    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:50:29.975095 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972959    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:50:29.975095 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972963    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:50:29.975095 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972966    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:50:29.975095 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972969    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:50:29.975095 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972972    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:50:29.975095 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972975    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:50:29.975095 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972977    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:50:29.975095 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972980    2571 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:50:29.975095 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972982    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:50:29.975095 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972984    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:50:29.975095 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972987    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:50:29.975095 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972989    2571 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:50:29.975095 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972992    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:50:29.975095 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972994    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:50:29.975095 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.972999    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:50:29.975776 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973003    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:50:29.975776 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973006    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:50:29.975776 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973009    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:50:29.975776 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973011    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:50:29.975776 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973014    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:50:29.975776 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973016    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:50:29.975776 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973019    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:50:29.975776 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973021    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:50:29.975776 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973024    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:50:29.975776 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973026    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:50:29.975776 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973029    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:50:29.975776 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973031    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:50:29.975776 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973033    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:50:29.975776 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973036    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:50:29.975776 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973038    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:50:29.975776 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973040    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:50:29.975776 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973043    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:50:29.975776 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973488    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:50:29.975776 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973493    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:50:29.975776 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973496    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:50:29.976526 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973499    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:50:29.976526 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973502    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:50:29.976526 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973505    2571 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:50:29.976526 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973507    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:50:29.976526 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973510    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:50:29.976526 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973513    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:50:29.976526 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973515    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:50:29.976526 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973518    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:50:29.976526 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973520    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:50:29.976526 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973523    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:50:29.976526 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973525    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:50:29.976526 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973528    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:50:29.976526 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973530    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:50:29.976526 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973533    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:50:29.976526 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973535    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:50:29.976526 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973538    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:50:29.976526 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973540    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:50:29.976526 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973543    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:50:29.976526 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973545    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:50:29.977092 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973548    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:50:29.977092 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973550    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:50:29.977092 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973552    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:50:29.977092 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973555    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:50:29.977092 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973557    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:50:29.977092 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973560    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:50:29.977092 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973562    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:50:29.977092 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973564    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:50:29.977092 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973567    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:50:29.977092 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973570    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:50:29.977092 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973574    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:50:29.977092 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973578    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:50:29.977092 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973583    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:50:29.977092 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973586    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:50:29.977092 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973589    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:50:29.977092 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973592    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:50:29.977092 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973594    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:50:29.977092 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973596    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:50:29.977518 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973599    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:50:29.977518 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973601    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:50:29.977518 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973604    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:50:29.977518 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973607    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:50:29.977518 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973609    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:50:29.977518 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973611    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:50:29.977518 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973614    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:50:29.977518 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973617    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:50:29.977518 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973619    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:50:29.977518 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973630    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:50:29.977518 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973633    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:50:29.977518 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973635    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:50:29.977518 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973638    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:50:29.977518 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973640    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:50:29.977518 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973643    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:50:29.977518 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973645    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:50:29.977518 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973648    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:50:29.977518 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973650    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:50:29.977518 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973653    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:50:29.977518 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973656    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:50:29.978045 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973676    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:50:29.978045 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973679    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:50:29.978045 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973682    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:50:29.978045 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973685    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:50:29.978045 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973687    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:50:29.978045 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973704    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:50:29.978045 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973708    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:50:29.978045 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973711    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:50:29.978045 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973714    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:50:29.978045 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973716    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:50:29.978045 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973720    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:50:29.978045 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973722    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:50:29.978045 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973725    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:50:29.978045 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973728    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:50:29.978045 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973730    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:50:29.978045 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973733    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:50:29.978045 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973735    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:50:29.978045 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973738    2571 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:50:29.978045 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973740    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:50:29.978045 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973743    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:50:29.978514 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973745    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:50:29.978514 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973748    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:50:29.978514 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973751    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:50:29.978514 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973753    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:50:29.978514 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973756    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:50:29.978514 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973758    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:50:29.978514 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973761    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:50:29.978514 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973763    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:50:29.978514 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973766    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:50:29.978514 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973768    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:50:29.978514 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973771    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:50:29.978514 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973773    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:50:29.978514 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973776    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:50:29.978514 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973778    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:50:29.978514 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973781    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:50:29.978514 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973784    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:50:29.978514 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973787    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:50:29.978514 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973790    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:50:29.978514 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973793    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:50:29.978514 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973796    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973798    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973801    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973803    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973806    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973809    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.973811    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974657    2571 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974666    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974673    2571 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974677    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974683    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974686    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974702    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974707    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974710    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974713    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974717    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974720    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974723    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974726    2571 flags.go:64] FLAG: --cgroup-root=""
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974729    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974732    2571 flags.go:64] FLAG: --client-ca-file=""
Nov 03 01:50:29.978997 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974735    2571 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974738    2571 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974743    2571 flags.go:64] FLAG: --cluster-domain=""
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974746    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974749    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974752    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974755    2571 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974759    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974763    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974766    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974770    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974773    2571 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974776    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974779    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974782    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974785    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974790    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974793    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974796    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974799    2571 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974802    2571 flags.go:64] FLAG: --enable-server="true"
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974805    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974809    2571 flags.go:64] FLAG: --event-burst="100"
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974812    2571 flags.go:64] FLAG: --event-qps="50"
Nov 03 01:50:29.979510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974815    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974819    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974826    2571 flags.go:64] FLAG: --eviction-hard=""
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974830    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974833    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974836    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974839    2571 flags.go:64] FLAG: --eviction-soft=""
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974842    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974845    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974848    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974851    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974854    2571 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974858    2571 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974874    2571 flags.go:64] FLAG: --feature-gates=""
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974877    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974881    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974884    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974887    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974890    2571 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974893    2571 flags.go:64] FLAG: --help="false"
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974896    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974899    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974902    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 01:50:29.980160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974906    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974909    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974913    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974916    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974918    2571 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974921    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974924    2571 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974927    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974930    2571 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974933    2571 flags.go:64] FLAG: --kube-reserved=""
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974936    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974940    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974943    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974946    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974949    2571 flags.go:64] FLAG: --lock-file=""
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974952    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974955    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974958    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974963    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974966    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974969    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974972    2571 flags.go:64] FLAG: --logging-format="text"
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974975    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974979    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 01:50:29.980724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974981    2571 flags.go:64] FLAG: --manifest-url=""
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974984    2571 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974989    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974992    2571 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974995    2571 flags.go:64] FLAG: --max-pods="110"
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.974999    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975002    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975005    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975008    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975011    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975014    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975017    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975023    2571 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975027    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975030    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975033    2571 flags.go:64] FLAG: --pod-cidr=""
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975036    2571 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975038    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975041    2571 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975046    2571 flags.go:64] FLAG: --port="10250"
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975049    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975051    2571 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-03c7b893505f44a8c"
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975055    2571 flags.go:64] FLAG: --qos-reserved=""
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975057    2571 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975062    2571 flags.go:64] FLAG: --register-node="true"
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975065    2571 flags.go:64] FLAG: --register-with-taints=""
Nov 03 01:50:29.981284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975069    2571 flags.go:64] FLAG: --registry-burst="10"
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975072    2571 flags.go:64] FLAG: --registry-qps="5"
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975075    2571 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975078    2571 flags.go:64] FLAG: --reserved-memory=""
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975082    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975085    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975088    2571 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975091    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975093    2571 flags.go:64] FLAG: --runonce="false"
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975096    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975100    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975103    2571 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975106    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975109    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975113    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975116    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975119    2571 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975123    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975126    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975128    2571 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975131    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975134    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975137    2571 flags.go:64] FLAG: --system-cgroups=""
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975139    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975145    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 01:50:29.981931 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975149    2571 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975152    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975155    2571 flags.go:64] FLAG: --tls-min-version=""
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975158    2571 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975161    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975165    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975168    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975171    2571 flags.go:64] FLAG: --v="2"
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975175    2571 flags.go:64] FLAG: --version="false"
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975179    2571 flags.go:64] FLAG: --vmodule=""
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975183    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.975186    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976107    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976125    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976131    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976135    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976139    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976143    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976148    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976152    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976156    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976160    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976164    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:50:29.982617 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976169    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:50:29.983197 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976176    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:50:29.983197 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976185    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:50:29.983197 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976190    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:50:29.983197 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976197    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:50:29.983197 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976203    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:50:29.983197 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976207    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:50:29.983197 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976212    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:50:29.983197 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976217    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:50:29.983197 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976222    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:50:29.983197 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976226    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:50:29.983197 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976229    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:50:29.983197 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976233    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:50:29.983197 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976237    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:50:29.983197 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976249    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:50:29.983197 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976253    2571 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:50:29.983197 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976257    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:50:29.983197 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976261    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:50:29.983197 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976265    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:50:29.983197 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976269    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:50:29.983643 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976273    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:50:29.983643 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976277    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:50:29.983643 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976281    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:50:29.983643 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976285    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:50:29.983643 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976289    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:50:29.983643 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976292    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:50:29.983643 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976296    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:50:29.983643 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976305    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:50:29.983643 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976308    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:50:29.983643 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976312    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:50:29.983643 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976316    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:50:29.983643 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976320    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:50:29.983643 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976324    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:50:29.983643 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976329    2571 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:50:29.983643 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976333    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:50:29.983643 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976338    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:50:29.983643 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976342    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:50:29.983643 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976346    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:50:29.983643 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976350    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:50:29.984086 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976361    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:50:29.984086 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976366    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:50:29.984086 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976371    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:50:29.984086 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976375    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:50:29.984086 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976417    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:50:29.984086 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976428    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:50:29.984086 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976433    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:50:29.984086 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976453    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:50:29.984086 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976578    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:50:29.984086 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976583    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:50:29.984086 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976586    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:50:29.984086 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976589    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:50:29.984086 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976592    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:50:29.984086 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976596    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:50:29.984086 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976598    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:50:29.984086 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976601    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:50:29.984086 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976604    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:50:29.984086 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976606    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:50:29.984086 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976609    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:50:29.984523 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976612    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:50:29.984523 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976615    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:50:29.984523 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976617    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:50:29.984523 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976620    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:50:29.984523 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976622    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:50:29.984523 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976625    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:50:29.984523 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976628    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:50:29.984523 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976631    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:50:29.984523 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976633    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:50:29.984523 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976636    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:50:29.984523 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976638    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:50:29.984523 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976641    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:50:29.984523 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976647    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:50:29.984523 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976651    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:50:29.984523 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976654    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:50:29.984523 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976656    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:50:29.984523 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976659    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:50:29.984523 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976662    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:50:29.984523 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976665    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:50:29.984523 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976667    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:50:29.985038 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976671    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:50:29.985038 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976673    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:50:29.985038 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976676    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:50:29.985038 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976680    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:50:29.985038 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976683    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:50:29.985038 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976686    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:50:29.985038 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976688    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:50:29.985038 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976707    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:50:29.985038 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976710    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:50:29.985038 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976713    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:50:29.985038 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976715    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:50:29.985038 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976719    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:50:29.985038 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976722    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:50:29.985038 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976724    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:50:29.985038 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976727    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:50:29.985038 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976730    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:50:29.985038 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976734    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:50:29.985437 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.976774    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 01:50:29.985437 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976823    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:50:29.985437 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976829    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:50:29.985437 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976834    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:50:29.985437 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976837    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:50:29.985437 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976839    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:50:29.985437 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976844    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:50:29.985437 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976847    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:50:29.985437 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976851    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:50:29.985437 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976854    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:50:29.985437 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976857    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:50:29.985437 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976860    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:50:29.985437 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976862    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:50:29.985437 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976865    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:50:29.985437 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976867    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:50:29.985835 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976870    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:50:29.985835 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976873    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:50:29.985835 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976876    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:50:29.985835 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976878    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:50:29.985835 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976881    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:50:29.985835 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976883    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:50:29.985835 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976886    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:50:29.985835 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976888    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:50:29.985835 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976891    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:50:29.985835 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976893    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:50:29.985835 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976896    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:50:29.985835 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976899    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:50:29.985835 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976901    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:50:29.985835 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976903    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:50:29.985835 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976906    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:50:29.985835 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976908    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:50:29.985835 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976910    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:50:29.985835 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976913    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:50:29.985835 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976915    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:50:29.985835 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976918    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:50:29.986306 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976920    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:50:29.986306 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976923    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:50:29.986306 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976925    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:50:29.986306 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976928    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:50:29.986306 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976930    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:50:29.986306 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976933    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:50:29.986306 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976935    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:50:29.986306 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976938    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:50:29.986306 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976941    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:50:29.986306 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976943    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:50:29.986306 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976946    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:50:29.986306 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976948    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:50:29.986306 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976951    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:50:29.986306 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976954    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:50:29.986306 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976957    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:50:29.986306 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976959    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:50:29.986306 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976962    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:50:29.986306 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976964    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:50:29.986306 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976967    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:50:29.986306 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976969    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:50:29.986784 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976972    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:50:29.986784 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976975    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:50:29.986784 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976977    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:50:29.986784 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976980    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:50:29.986784 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976982    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:50:29.986784 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976984    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:50:29.986784 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976987    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:50:29.986784 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976989    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:50:29.986784 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976992    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:50:29.986784 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976994    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:50:29.986784 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976997    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:50:29.986784 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.976999    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:50:29.986784 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977002    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:50:29.986784 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977004    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:50:29.986784 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977007    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:50:29.986784 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977009    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:50:29.986784 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977011    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:50:29.986784 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977014    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:50:29.986784 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977017    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:50:29.986784 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977022    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:50:29.987255 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977025    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:50:29.987255 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977030    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:50:29.987255 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977034    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:50:29.987255 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977038    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:50:29.987255 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977041    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:50:29.987255 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977045    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:50:29.987255 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977048    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:50:29.987255 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977050    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:50:29.987255 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977053    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:50:29.987255 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977056    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:50:29.987255 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977058    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:50:29.987255 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977061    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:50:29.987255 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977063    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:50:29.987255 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977066    2571 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:50:29.987255 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977068    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:50:29.987255 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977071    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:50:29.987255 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977073    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:50:29.987255 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977076    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:50:29.987255 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977078    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:50:29.987689 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977081    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:50:29.987689 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977084    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:50:29.987689 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977086    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:50:29.987689 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977088    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:50:29.987689 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977091    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:50:29.987689 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977094    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:50:29.987689 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977096    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:50:29.987689 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977099    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:50:29.987689 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977101    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:50:29.987689 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977104    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:50:29.987689 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977107    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:50:29.987689 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977110    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:50:29.987689 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.977114    2571 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:50:29.987689 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.983718    2571 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 01:50:29.987689 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.983731    2571 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 01:50:29.987689 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.984470    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 01:50:29.987689 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.984476    2571 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 01:50:29.987689 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984531    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:50:29.987689 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984537    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:50:29.987689 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984541    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:50:29.988157 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984545    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:50:29.988157 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984548    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:50:29.988157 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984550    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:50:29.988157 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984553    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:50:29.988157 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984556    2571 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:50:29.988157 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984558    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:50:29.988157 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984561    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:50:29.988157 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984564    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:50:29.988157 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984566    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:50:29.988157 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984569    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:50:29.988157 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984572    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:50:29.988157 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984574    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:50:29.988157 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984577    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:50:29.988157 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984579    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:50:29.988157 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984581    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:50:29.988157 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984584    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:50:29.988157 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984587    2571 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:50:29.988157 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984589    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:50:29.988157 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984592    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:50:29.988157 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984594    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:50:29.988621 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984597    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:50:29.988621 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984600    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:50:29.988621 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984602    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:50:29.988621 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984605    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:50:29.988621 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984607    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:50:29.988621 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984610    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:50:29.988621 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984614    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:50:29.988621 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984617    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:50:29.988621 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984619    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:50:29.988621 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984622    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:50:29.988621 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984626    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:50:29.988621 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984630    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:50:29.988621 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984633    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:50:29.988621 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984637    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:50:29.988621 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984640    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:50:29.988621 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984642    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:50:29.988621 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984645    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:50:29.988621 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984648    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:50:29.988621 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984650    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:50:29.989084 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984653    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:50:29.989084 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984656    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:50:29.989084 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984658    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:50:29.989084 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984661    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:50:29.989084 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984664    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:50:29.989084 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984667    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:50:29.989084 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984669    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:50:29.989084 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984672    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:50:29.989084 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984675    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:50:29.989084 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984677    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:50:29.989084 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984680    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:50:29.989084 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984683    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:50:29.989084 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984685    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:50:29.989084 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984687    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:50:29.989084 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984703    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:50:29.989084 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984706    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:50:29.989084 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984719    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:50:29.989084 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984722    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:50:29.989084 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984725    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:50:29.989084 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984728    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:50:29.989555 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984731    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:50:29.989555 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984734    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:50:29.989555 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984737    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:50:29.989555 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984739    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:50:29.989555 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984742    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:50:29.989555 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984746    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:50:29.989555 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984750    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:50:29.989555 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984753    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:50:29.989555 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984756    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:50:29.989555 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984759    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:50:29.989555 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984761    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:50:29.989555 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984764    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:50:29.989555 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984766    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:50:29.989555 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984769    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:50:29.989555 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984771    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:50:29.989555 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984774    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:50:29.989555 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984776    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:50:29.989555 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984779    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:50:29.989555 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984781    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:50:29.990109 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984784    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:50:29.990109 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984786    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:50:29.990109 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984789    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:50:29.990109 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984791    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:50:29.990109 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984793    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:50:29.990109 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984796    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:50:29.990109 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984798    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:50:29.990109 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984801    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:50:29.990109 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984803    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:50:29.990109 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984806    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:50:29.990109 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984808    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:50:29.990109 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984811    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:50:29.990109 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984814    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:50:29.990109 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984817    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:50:29.990109 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984820    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:50:29.990109 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984823    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:50:29.990109 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984826    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:50:29.990109 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984828    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:50:29.990109 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984831    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:50:29.990564 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984833    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:50:29.990564 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984836    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:50:29.990564 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984838    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:50:29.990564 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984841    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:50:29.990564 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984843    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:50:29.990564 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.984846    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:50:29.990564 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.984873    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 01:50:29.990564 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985007    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:50:29.990564 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985012    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:50:29.990564 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985015    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:50:29.990564 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985018    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:50:29.990564 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985021    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:50:29.990564 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985024    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:50:29.990564 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985026    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:50:29.990564 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985029    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:50:29.990564 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985031    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:50:29.990956 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985034    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:50:29.990956 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985037    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:50:29.990956 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985039    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:50:29.990956 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985042    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:50:29.990956 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985044    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:50:29.990956 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985047    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:50:29.990956 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985050    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:50:29.990956 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985053    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:50:29.990956 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985055    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:50:29.990956 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985058    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:50:29.990956 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985061    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:50:29.990956 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985064    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:50:29.990956 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985068    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:50:29.990956 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985073    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:50:29.990956 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985076    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:50:29.990956 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985079    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:50:29.990956 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985081    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:50:29.990956 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985084    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:50:29.990956 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985086    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:50:29.992090 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985089    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:50:29.992090 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985091    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:50:29.992090 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985094    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:50:29.992090 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985096    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:50:29.992090 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985099    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:50:29.992090 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985102    2571 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:50:29.992090 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985104    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:50:29.992090 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985107    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:50:29.992090 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985109    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:50:29.992090 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985111    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:50:29.992090 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985114    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:50:29.992090 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985117    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:50:29.992090 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985119    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:50:29.992090 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985121    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:50:29.992090 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985124    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:50:29.992090 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985127    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:50:29.992090 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985129    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:50:29.992090 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985132    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:50:29.992090 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985134    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:50:29.992090 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985137    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:50:29.992556 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985139    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:50:29.992556 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985142    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:50:29.992556 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985145    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:50:29.992556 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985147    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:50:29.992556 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985150    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:50:29.992556 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985152    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:50:29.992556 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985155    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:50:29.992556 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985158    2571 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:50:29.992556 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985160    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:50:29.992556 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985163    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:50:29.992556 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985165    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:50:29.992556 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985167    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:50:29.992556 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985170    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:50:29.992556 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985172    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:50:29.992556 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985175    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:50:29.992556 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985177    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:50:29.992556 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985180    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:50:29.992556 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985182    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:50:29.992556 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985185    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:50:29.992556 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985187    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:50:29.993041 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985189    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:50:29.993041 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985192    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:50:29.993041 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985194    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:50:29.993041 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985197    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:50:29.993041 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985199    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:50:29.993041 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985201    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:50:29.993041 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985204    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:50:29.993041 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985207    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:50:29.993041 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985209    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:50:29.993041 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985211    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:50:29.993041 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985214    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:50:29.993041 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985216    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:50:29.993041 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985219    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:50:29.993041 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985221    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:50:29.993041 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985224    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:50:29.993041 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985228    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:50:29.993041 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985231    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:50:29.993041 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985234    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:50:29.993041 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985236    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:50:29.993480 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985239    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:50:29.993480 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985242    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:50:29.993480 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985245    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:50:29.993480 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985247    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:50:29.993480 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985250    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:50:29.993480 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985252    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:50:29.993480 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985254    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:50:29.993480 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985257    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:50:29.993480 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985260    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:50:29.993480 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985262    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:50:29.993480 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985264    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:50:29.993480 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985267    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:50:29.993480 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985269    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:50:29.993480 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985272    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:50:29.993480 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985274    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:50:29.993480 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985276    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:50:29.993480 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985279    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:50:29.993480 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985281    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:50:29.993480 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985284    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:50:29.994113 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.985305    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 01:50:29.994113 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985337    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:50:29.994113 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985341    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:50:29.994113 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985345    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:50:29.994113 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985347    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:50:29.994113 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985350    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:50:29.994113 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985353    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:50:29.994113 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985355    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:50:29.994113 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985358    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:50:29.994113 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985361    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:50:29.994113 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985364    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:50:29.994113 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985366    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:50:29.994113 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985369    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:50:29.994113 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985371    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:50:29.994113 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985374    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:50:29.994113 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985377    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:50:29.994487 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985380    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:50:29.994487 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985382    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:50:29.994487 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985385    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:50:29.994487 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985387    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:50:29.994487 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985390    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:50:29.994487 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985392    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:50:29.994487 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985395    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:50:29.994487 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985397    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:50:29.994487 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985400    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:50:29.994487 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985403    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:50:29.994487 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985405    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:50:29.994487 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985408    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:50:29.994487 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985410    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:50:29.994487 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985413    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:50:29.994487 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985415    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:50:29.994487 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985421    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:50:29.994487 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985423    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:50:29.994487 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985425    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:50:29.994487 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985428    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:50:29.994953 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985431    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:50:29.994953 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985433    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:50:29.994953 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985436    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:50:29.994953 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985438    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:50:29.994953 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985441    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:50:29.994953 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985444    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:50:29.994953 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985448    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:50:29.994953 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985452    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:50:29.994953 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985455    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:50:29.994953 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985459    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:50:29.994953 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985461    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:50:29.994953 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985464    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:50:29.994953 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985467    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:50:29.994953 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985469    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:50:29.994953 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985472    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:50:29.994953 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985474    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:50:29.994953 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985477    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:50:29.994953 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985479    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:50:29.994953 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985482    2571 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:50:29.995371 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985484    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:50:29.995371 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985487    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:50:29.995371 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985489    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:50:29.995371 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985491    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:50:29.995371 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985494    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:50:29.995371 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985496    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:50:29.995371 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985500    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:50:29.995371 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985502    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:50:29.995371 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985504    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:50:29.995371 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985508    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:50:29.995371 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985511    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:50:29.995371 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985514    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:50:29.995371 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985516    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:50:29.995371 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985518    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:50:29.995371 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985521    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:50:29.995371 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985523    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:50:29.995371 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985526    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:50:29.995371 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985528    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:50:29.995371 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985531    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:50:29.995371 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985533    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:50:29.995830 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985536    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:50:29.995830 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985539    2571 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:50:29.995830 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985541    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:50:29.995830 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985544    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:50:29.995830 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985546    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:50:29.995830 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985549    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:50:29.995830 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985552    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:50:29.995830 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985555    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:50:29.995830 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985557    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:50:29.995830 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985560    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:50:29.995830 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985562    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:50:29.995830 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985565    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:50:29.995830 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985567    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:50:29.995830 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985570    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:50:29.995830 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985572    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:50:29.995830 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985575    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:50:29.995830 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985577    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:50:29.995830 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985579    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:50:29.995830 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985582    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:50:29.995830 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985584    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:50:29.996296 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985587    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:50:29.996296 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985591    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:50:29.996296 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985593    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:50:29.996296 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985596    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:50:29.996296 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985598    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:50:29.996296 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985601    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:50:29.996296 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985603    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:50:29.996296 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985605    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:50:29.996296 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985608    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:50:29.996296 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985612    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:50:29.996296 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985616    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:50:29.996296 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985619    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:50:29.996296 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:29.985630    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:50:29.996296 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.985724    2571 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 01:50:29.996296 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.987972    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 01:50:29.996296 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.988909    2571 server.go:1014] "Starting client certificate rotation"
Nov 03 01:50:29.996296 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.989002    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 01:50:29.996296 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.989030    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 01:50:29.996732 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.990854    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 01:50:29.996732 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:29.991921    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 01:50:30.002810 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.002793    2571 log.go:25] "Validated CRI v1 runtime API"
Nov 03 01:50:30.007140 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.006411    2571 log.go:25] "Validated CRI v1 image API"
Nov 03 01:50:30.008050 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.008026    2571 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 01:50:30.012665 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.012645    2571 fs.go:135] Filesystem UUIDs: map[557a8415-5443-4dfd-8bac-4442f77ff790:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e4ed0fdc-bd28-4343-a9fa-a720d63f539b:/dev/nvme0n1p3]
Nov 03 01:50:30.012756 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.012663    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 01:50:30.017679 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.017550    2571 manager.go:217] Machine: {Timestamp:2026-11-03 01:50:30.016213059 +0000 UTC m=+0.303783845 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3187370 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d7708420d8838b64bd98fb15f3304 SystemUUID:ec2d7708-420d-8838-b64b-d98fb15f3304 BootID:2bfd3b4a-86b6-4a98-bf0f-108cf1122432 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:ec:62:96:59 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:ec:62:96:59 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:cc:f4:24:e1:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 01:50:30.017679 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.017677    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 01:50:30.017814 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.017801    2571 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 01:50:30.020923 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.020901    2571 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 01:50:30.021075 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.020924    2571 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-26.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 01:50:30.021769 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.021757    2571 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 01:50:30.021823 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.021774    2571 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 01:50:30.021823 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.021813    2571 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 01:50:30.021924 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.021833    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 01:50:30.022772 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.022760    2571 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 01:50:30.026907 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.026893    2571 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 01:50:30.027038 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.027027    2571 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 01:50:30.027746 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.027736    2571 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 01:50:30.027799 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.027753    2571 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 01:50:30.027799 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.027768    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 01:50:30.027799 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.027781    2571 kubelet.go:405] "Adding apiserver pod source"
Nov 03 01:50:30.027799 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.027792    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 01:50:30.028165 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.028135    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 01:50:30.030046 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.030027    2571 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 01:50:30.035454 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.035441    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 01:50:30.036217 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.036205    2571 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 01:50:30.036262 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.036241    2571 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 01:50:30.036390 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.036381    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 01:50:30.036420 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.036396    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 01:50:30.036420 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.036402    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 01:50:30.036420 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.036408    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 01:50:30.036420 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.036415    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 01:50:30.036420 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.036420    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 01:50:30.036903 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.036895    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 01:50:30.036933 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.036904    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 01:50:30.036933 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.036912    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 01:50:30.036933 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.036918    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 01:50:30.036933 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.036932    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 01:50:30.037122 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.036940    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 01:50:30.037654 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.037644    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 01:50:30.037654 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.037653    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 01:50:30.041429 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.041416    2571 server.go:1263] "Started kubelet"
Nov 03 01:50:30.041571 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.041522    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 01:50:30.041613 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.041545    2571 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 01:50:30.041613 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.041591    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 01:50:30.042171 ip-10-0-139-26 systemd[1]: Started Kubernetes Kubelet.
Nov 03 01:50:30.042803 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.042783    2571 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 01:50:30.044305 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.043994    2571 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 01:50:30.049984 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:30.049965    2571 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 01:50:30.050241 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.050222    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 01:50:30.050877 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.050853    2571 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 01:50:30.051135 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:30.051116    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-26.us-west-2.compute.internal\" not found"
Nov 03 01:50:30.051561 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.051544    2571 factory.go:55] Registering systemd factory
Nov 03 01:50:30.051561 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.051560    2571 factory.go:223] Registration of the systemd container factory successfully
Nov 03 01:50:30.051733 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.051573    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 01:50:30.051733 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.051576    2571 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 01:50:30.051733 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.051594    2571 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 01:50:30.051733 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.051681    2571 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 01:50:30.051733 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.051689    2571 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 01:50:30.051991 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.051770    2571 factory.go:153] Registering CRI-O factory
Nov 03 01:50:30.051991 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.051787    2571 factory.go:223] Registration of the crio container factory successfully
Nov 03 01:50:30.051991 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.051850    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 01:50:30.051991 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.051868    2571 factory.go:103] Registering Raw factory
Nov 03 01:50:30.051991 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.051884    2571 manager.go:1199] Started watching for new ooms in manager
Nov 03 01:50:30.053004 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.052988    2571 manager.go:319] Starting recovery of all containers
Nov 03 01:50:30.065094 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.065075    2571 manager.go:324] Recovery completed
Nov 03 01:50:30.069401 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.069388    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:50:30.071267 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.071240    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:50:30.071338 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.071283    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:50:30.071338 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.071299    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:50:30.071797 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.071780    2571 cpu_manager.go:226] "Starting" policy="none"
Nov 03 01:50:30.071797 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.071796    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:50:30.071944 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.071795    2571 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 01:50:30.071944 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.071864    2571 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 01:50:30.072672 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.072652    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:50:30.072753 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.072680    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:50:30.072790 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.072758    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:50:30.074132 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.074119    2571 policy_none.go:50] "Start"
Nov 03 01:50:30.074178 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.074135    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 01:50:30.074178 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.074148    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 01:50:30.076131 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.076118    2571 policy_none.go:44] "Start"
Nov 03 01:50:30.126933 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.120402    2571 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 01:50:30.126933 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.123758    2571 manager.go:341] "Starting Device Plugin manager"
Nov 03 01:50:30.126933 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:30.124317    2571 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 01:50:30.126933 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.124345    2571 server.go:87] "Starting device plugin registration server"
Nov 03 01:50:30.126933 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.124607    2571 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 01:50:30.126933 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.124619    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 01:50:30.126933 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.124721    2571 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 01:50:30.126933 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.124811    2571 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 01:50:30.126933 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.124822    2571 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 01:50:30.126933 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:30.125432    2571 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 01:50:30.126933 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:30.125474    2571 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-26.us-west-2.compute.internal\" not found"
Nov 03 01:50:30.141855 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.141839    2571 container_manager_linux.go:682] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2608 into the system container \"/system.slice\": <nil>"
Nov 03 01:50:30.180385 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.180359    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-26.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 01:50:30.180474 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:30.180359    2571 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-26.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 01:50:30.227074 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.226898    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:50:30.227799 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:30.226895    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-26.us-west-2.compute.internal.18e465ef4bab0054  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-26.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-26.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 01:50:30.041387092 +0000 UTC m=+0.328957877,LastTimestamp:2026-11-03 01:50:30.041387092 +0000 UTC m=+0.328957877,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-26.us-west-2.compute.internal,}"
Nov 03 01:50:30.227907 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.227879    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:50:30.227907 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.227905    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:50:30.228355 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.228344    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:50:30.228398 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.228389    2571 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:50:30.239817 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.239792    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 01:50:30.240957 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.240934    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 01:50:30.241027 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.240961    2571 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 01:50:30.241027 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.240996    2571 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 01:50:30.241088 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:30.241034    2571 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 01:50:30.268496 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.268477    2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 01:50:30.268582 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.268536    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mbfln"
Nov 03 01:50:30.341599 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.341555    2571 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal"]
Nov 03 01:50:30.341778 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.341640    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:50:30.342583 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.342569    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:50:30.342636 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.342598    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:50:30.342636 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.342614    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:50:30.344803 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.344792    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:50:30.344960 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.344946    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:50:30.345012 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.345001    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:50:30.345505 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.345485    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:50:30.345557 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.345517    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:50:30.345557 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.345533    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:50:30.345640 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.345580    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:50:30.345640 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.345599    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:50:30.345640 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.345608    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:50:30.346171 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.346157    2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 01:50:30.347379 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.347365    2571 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 01:50:30.347783 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.347768    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:50:30.347827 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.347799    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:50:30.348460 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.348446    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:50:30.348543 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.348474    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:50:30.348543 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.348488    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:50:30.353781 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.353764    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6076c2cc730e7ded3ca7127f0bb3e92b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal\" (UID: \"6076c2cc730e7ded3ca7127f0bb3e92b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:50:30.353844 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.353789    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6076c2cc730e7ded3ca7127f0bb3e92b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal\" (UID: \"6076c2cc730e7ded3ca7127f0bb3e92b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:50:30.353844 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.353806    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37711de5beea2605bd7d0ac9a465c416-config\") pod \"kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal\" (UID: \"37711de5beea2605bd7d0ac9a465c416\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:50:30.376199 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:30.376183    2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-26.us-west-2.compute.internal\" not found" node="ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:50:30.378596 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:30.378582    2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-26.us-west-2.compute.internal\" not found" node="ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:50:30.449817 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.449797    2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 01:50:30.450337 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.450323    2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 01:50:30.451314 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.451302    2571 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:50:30.451361 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:30.451319    2571 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-26.us-west-2.compute.internal\": node \"ip-10-0-139-26.us-west-2.compute.internal\" not found"
Nov 03 01:50:30.451398 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.451369    2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:50:30.454798 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.454786    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6076c2cc730e7ded3ca7127f0bb3e92b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal\" (UID: \"6076c2cc730e7ded3ca7127f0bb3e92b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:50:30.454870 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.454807    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6076c2cc730e7ded3ca7127f0bb3e92b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal\" (UID: \"6076c2cc730e7ded3ca7127f0bb3e92b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:50:30.454870 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.454822    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37711de5beea2605bd7d0ac9a465c416-config\") pod \"kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal\" (UID: \"37711de5beea2605bd7d0ac9a465c416\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:50:30.454934 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.454877    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37711de5beea2605bd7d0ac9a465c416-config\") pod \"kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal\" (UID: \"37711de5beea2605bd7d0ac9a465c416\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:50:30.454934 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.454898    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6076c2cc730e7ded3ca7127f0bb3e92b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal\" (UID: \"6076c2cc730e7ded3ca7127f0bb3e92b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:50:30.454934 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.454902    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6076c2cc730e7ded3ca7127f0bb3e92b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal\" (UID: \"6076c2cc730e7ded3ca7127f0bb3e92b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:50:30.567675 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.567620    2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 01:50:30.567901 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.567884    2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:50:30.667116 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.667085    2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 01:50:30.680271 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.680247    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:50:30.682423 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.682406    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:50:30.989290 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.989206    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 01:50:30.989889 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.989329    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 01:50:30.989889 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.989369    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 01:50:30.989889 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:30.989386    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 01:50:30.989889 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:30.989402    2571 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ab176cf1b8f3e4b2b9a47658d610d089-75b99088ee9799d1.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-139-26.us-west-2.compute.internal.18e465ef4d738170\": read tcp 10.0.139.26:44094->3.128.139.37:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-139-26.us-west-2.compute.internal.18e465ef4d738170  default   4862 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-26.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-26.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-26.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 01:50:30 +0000 UTC,LastTimestamp:2026-11-03 01:50:30.228353865 +0000 UTC m=+0.515924654,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-26.us-west-2.compute.internal,}"
Nov 03 01:50:31.028195 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.028160    2571 apiserver.go:52] "Watching apiserver"
Nov 03 01:50:31.051130 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.051112    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 01:50:31.274171 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.274082    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 01:45:30 +0000 UTC" deadline="2028-05-30 14:15:53.242476041 +0000 UTC"
Nov 03 01:50:31.274171 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.274115    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13788h25m21.968365266s"
Nov 03 01:50:31.320071 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.320046    2571 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 01:50:31.320387 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.320369    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-mxg52","openshift-cluster-node-tuning-operator/tuned-25wbg","openshift-image-registry/node-ca-qg2kq","openshift-multus/multus-9kq9k","openshift-multus/network-metrics-daemon-xv8zr","openshift-network-operator/iptables-alerter-tp5hd","kube-system/konnectivity-agent-vvzws","kube-system/kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs","openshift-dns/node-resolver-p8kgq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-mdr8p","openshift-multus/multus-additional-cni-plugins-d7qsp","openshift-ovn-kubernetes/ovnkube-node-jdk59"]
Nov 03 01:50:31.323106 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.323089    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.325291 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.325265    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.328096 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.328074    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.330903 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.330883    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.333173 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.333158    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:31.333241 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:31.333225    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv8zr" podUID="c7a013c2-18a4-4372-adcb-8dfafa673f9c"
Nov 03 01:50:31.335465 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.335433    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.337732 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.337716    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vvzws"
Nov 03 01:50:31.340068 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.340054    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:31.340117 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:31.340102    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mxg52" podUID="79fe02fa-99ce-4222-b569-6b323df47ab4"
Nov 03 01:50:31.342326 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.342309    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p8kgq"
Nov 03 01:50:31.344654 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.344636    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:50:31.344781 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:31.344752    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdr8p" podUID="42a56b4b-283d-4e63-bb6a-59c5d5df0c2c"
Nov 03 01:50:31.347117 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.347093    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tp5hd"
Nov 03 01:50:31.349481 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.349462    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qg2kq"
Nov 03 01:50:31.352545 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.352526    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 01:50:31.360040 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360018    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e49068d1-9ae4-47b3-9a07-3d695a0a2402-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.360126 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360044    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-hostroot\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.360126 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360062    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9260d15b-9772-4cb4-8588-8e82f89f39b8-serviceca\") pod \"node-ca-qg2kq\" (UID: \"9260d15b-9772-4cb4-8588-8e82f89f39b8\") " pod="openshift-image-registry/node-ca-qg2kq"
Nov 03 01:50:31.360126 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360077    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-slash\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.360126 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360095    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-run-ovn\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.360126 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360124    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79fe02fa-99ce-4222-b569-6b323df47ab4-dbus\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:31.360380 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360166    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-socket-dir-parent\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.360380 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360181    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-var-lib-cni-bin\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.360380 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360196    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6lhl\" (UniqueName: \"kubernetes.io/projected/8684a429-234f-4537-917e-5b49509bbdb5-kube-api-access-h6lhl\") pod \"node-resolver-p8kgq\" (UID: \"8684a429-234f-4537-917e-5b49509bbdb5\") " pod="openshift-dns/node-resolver-p8kgq"
Nov 03 01:50:31.360380 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360211    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-registration-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.360380 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360226    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-modprobe-d\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.360380 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360274    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-host\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.360380 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360353    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2d5x\" (UniqueName: \"kubernetes.io/projected/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-kube-api-access-f2d5x\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.360714 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360387    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e49068d1-9ae4-47b3-9a07-3d695a0a2402-cnibin\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.360714 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360423    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79fe02fa-99ce-4222-b569-6b323df47ab4-kubelet-config\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:31.360714 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360454    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b57cc991-d758-435f-9238-0dac97f10a93-host-slash\") pod \"iptables-alerter-tp5hd\" (UID: \"b57cc991-d758-435f-9238-0dac97f10a93\") " pod="openshift-network-operator/iptables-alerter-tp5hd"
Nov 03 01:50:31.360714 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360493    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-run-openvswitch\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.360714 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360524    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8684a429-234f-4537-917e-5b49509bbdb5-tmp-dir\") pod \"node-resolver-p8kgq\" (UID: \"8684a429-234f-4537-917e-5b49509bbdb5\") " pod="openshift-dns/node-resolver-p8kgq"
Nov 03 01:50:31.360714 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360554    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.360714 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360580    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-var-lib-kubelet\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.360714 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360604    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-tuned\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.360714 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360641    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.360714 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360665    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-env-overrides\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.360714 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360681    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-cni-binary-copy\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.360714 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360713    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-run-netns\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.361158 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360729    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h48h\" (UniqueName: \"kubernetes.io/projected/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-kube-api-access-5h48h\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.361158 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360746    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e49068d1-9ae4-47b3-9a07-3d695a0a2402-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.361158 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360763    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9260d15b-9772-4cb4-8588-8e82f89f39b8-host\") pod \"node-ca-qg2kq\" (UID: \"9260d15b-9772-4cb4-8588-8e82f89f39b8\") " pod="openshift-image-registry/node-ca-qg2kq"
Nov 03 01:50:31.361158 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360778    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-device-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.361158 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360814    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-systemd\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.361158 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360849    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-sys\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.361158 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360867    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flcc2\" (UniqueName: \"kubernetes.io/projected/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-kube-api-access-flcc2\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.361158 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360925    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e49068d1-9ae4-47b3-9a07-3d695a0a2402-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.361158 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360943    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-ovn-node-metrics-cert\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.361158 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360968    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e49068d1-9ae4-47b3-9a07-3d695a0a2402-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.361158 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.360987    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-run\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.361158 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361002    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-system-cni-dir\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.361158 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361019    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-sys-fs\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.361158 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361039    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-run-netns\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.361158 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361073    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f0923bb-db25-4266-ad8d-2e1cfb7f602b-agent-certs\") pod \"konnectivity-agent-vvzws\" (UID: \"1f0923bb-db25-4266-ad8d-2e1cfb7f602b\") " pod="kube-system/konnectivity-agent-vvzws"
Nov 03 01:50:31.361158 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361096    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-cni-dir\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.361754 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361112    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-cnibin\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.361754 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361149    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-run-k8s-cni-cncf-io\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.361754 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361208    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-daemon-config\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.361754 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361229    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-systemd-units\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.361754 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361247    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-log-socket\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.361754 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361272    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f0923bb-db25-4266-ad8d-2e1cfb7f602b-konnectivity-ca\") pod \"konnectivity-agent-vvzws\" (UID: \"1f0923bb-db25-4266-ad8d-2e1cfb7f602b\") " pod="kube-system/konnectivity-agent-vvzws"
Nov 03 01:50:31.361754 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361296    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-var-lib-cni-multus\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.361754 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361351    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-run-multus-certs\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.361754 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361377    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.361754 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361395    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqqkf\" (UniqueName: \"kubernetes.io/projected/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-kube-api-access-qqqkf\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.361754 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361411    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-sysctl-d\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.361754 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361440    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-sysctl-conf\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.361754 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361454    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-run-systemd\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.361754 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361468    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-run-ovn-kubernetes\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.361754 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361483    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e49068d1-9ae4-47b3-9a07-3d695a0a2402-system-cni-dir\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.361754 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361507    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcpqh\" (UniqueName: \"kubernetes.io/projected/e49068d1-9ae4-47b3-9a07-3d695a0a2402-kube-api-access-jcpqh\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.362359 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361527    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-conf-dir\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.362359 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361545    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-etc-kubernetes\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.362359 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361559    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhg5z\" (UniqueName: \"kubernetes.io/projected/9260d15b-9772-4cb4-8588-8e82f89f39b8-kube-api-access-hhg5z\") pod \"node-ca-qg2kq\" (UID: \"9260d15b-9772-4cb4-8588-8e82f89f39b8\") " pod="openshift-image-registry/node-ca-qg2kq"
Nov 03 01:50:31.362359 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361572    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-tmp\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.362359 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361589    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-ovnkube-script-lib\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.362359 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361621    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b57cc991-d758-435f-9238-0dac97f10a93-iptables-alerter-script\") pod \"iptables-alerter-tp5hd\" (UID: \"b57cc991-d758-435f-9238-0dac97f10a93\") " pod="openshift-network-operator/iptables-alerter-tp5hd"
Nov 03 01:50:31.362359 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361649    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-var-lib-kubelet\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.362359 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361678    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-kubelet\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.362359 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361728    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-etc-openvswitch\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.362359 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361748    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-node-log\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.362359 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361762    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-cni-netd\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.362359 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361783    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-ovnkube-config\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.362359 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361815    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq5lq\" (UniqueName: \"kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq\") pod \"network-check-target-mdr8p\" (UID: \"42a56b4b-283d-4e63-bb6a-59c5d5df0c2c\") " pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:50:31.362359 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361844    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs\") pod \"network-metrics-daemon-xv8zr\" (UID: \"c7a013c2-18a4-4372-adcb-8dfafa673f9c\") " pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:31.362359 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361861    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-sysconfig\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.362359 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361886    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-kubernetes\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.362807 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361909    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:31.362807 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361943    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltkx2\" (UniqueName: \"kubernetes.io/projected/c7a013c2-18a4-4372-adcb-8dfafa673f9c-kube-api-access-ltkx2\") pod \"network-metrics-daemon-xv8zr\" (UID: \"c7a013c2-18a4-4372-adcb-8dfafa673f9c\") " pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:31.362807 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.361962    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8684a429-234f-4537-917e-5b49509bbdb5-hosts-file\") pod \"node-resolver-p8kgq\" (UID: \"8684a429-234f-4537-917e-5b49509bbdb5\") " pod="openshift-dns/node-resolver-p8kgq"
Nov 03 01:50:31.362807 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.362007    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl96f\" (UniqueName: \"kubernetes.io/projected/b57cc991-d758-435f-9238-0dac97f10a93-kube-api-access-sl96f\") pod \"iptables-alerter-tp5hd\" (UID: \"b57cc991-d758-435f-9238-0dac97f10a93\") " pod="openshift-network-operator/iptables-alerter-tp5hd"
Nov 03 01:50:31.362807 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.362041    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-socket-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.362807 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.362067    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-lib-modules\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.362807 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.362092    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-var-lib-openvswitch\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.362807 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.362116    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-cni-bin\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.380315 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.380290    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 01:50:31.380396 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.380325    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 01:50:31.380396 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.380383    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 01:50:31.380608 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.380584    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pc8bh\""
Nov 03 01:50:31.382500 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.382357    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 01:50:31.382500 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.382435    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jwmm7\""
Nov 03 01:50:31.382641 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.382525    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 01:50:31.384924 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.384907    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 01:50:31.384984 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.384910    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 01:50:31.385254 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.385235    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 01:50:31.385297 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.385240    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 01:50:31.385297 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.385279    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 01:50:31.385571 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.385557    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7cgvv\""
Nov 03 01:50:31.385676 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.385664    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 01:50:31.387237 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.387224    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 01:50:31.387294 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.387247    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 01:50:31.387532 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.387516    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zq5dk\""
Nov 03 01:50:31.387588 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.387549    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 01:50:31.387588 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.387551    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 01:50:31.391866 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.391852    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 01:50:31.391947 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.391934    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 01:50:31.432193 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.432171    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 01:50:31.432316 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.432236    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 01:50:31.432316 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.432265    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hgghc\""
Nov 03 01:50:31.434892 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.434878    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 01:50:31.434958 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.434883    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 01:50:31.434958 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.434883    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5qdp4\""
Nov 03 01:50:31.435325 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.435309    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 01:50:31.435531 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.435309    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 01:50:31.435780 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.435758    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bqcwb\""
Nov 03 01:50:31.435867 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.435847    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 01:50:31.435977 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.435890    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2mmqt\""
Nov 03 01:50:31.436219 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.436191    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f7c55\""
Nov 03 01:50:31.436325 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.436284    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 01:50:31.436325 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.436311    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 01:50:31.436412 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.436383    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 01:50:31.436614 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.436589    2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 01:50:31.436744 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.436727    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lxs4j"
Nov 03 01:50:31.462532 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462513    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-sys-fs\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.462625 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462538    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-run-netns\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.462625 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462556    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f0923bb-db25-4266-ad8d-2e1cfb7f602b-agent-certs\") pod \"konnectivity-agent-vvzws\" (UID: \"1f0923bb-db25-4266-ad8d-2e1cfb7f602b\") " pod="kube-system/konnectivity-agent-vvzws"
Nov 03 01:50:31.462625 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462570    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-cni-dir\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.462625 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462584    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-cnibin\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.462625 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462601    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-run-k8s-cni-cncf-io\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.462625 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462615    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-daemon-config\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.462953 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462633    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-systemd-units\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.462953 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462635    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-run-netns\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.462953 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462658    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-log-socket\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.462953 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462664    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-cni-dir\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.462953 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462698    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-sys-fs\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.462953 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462694    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f0923bb-db25-4266-ad8d-2e1cfb7f602b-konnectivity-ca\") pod \"konnectivity-agent-vvzws\" (UID: \"1f0923bb-db25-4266-ad8d-2e1cfb7f602b\") " pod="kube-system/konnectivity-agent-vvzws"
Nov 03 01:50:31.462953 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462699    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-log-socket\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.462953 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462716    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-cnibin\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.462953 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462755    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-run-k8s-cni-cncf-io\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.462953 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462768    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-systemd-units\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.462953 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462782    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-var-lib-cni-multus\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.462953 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462810    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-run-multus-certs\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.462953 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462816    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-var-lib-cni-multus\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.462953 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462845    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.462953 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462855    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-run-multus-certs\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.462953 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462892    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqkf\" (UniqueName: \"kubernetes.io/projected/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-kube-api-access-qqqkf\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.462953 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462918    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-sysctl-d\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.462953 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462951    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-sysctl-conf\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.463670 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462969    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.463670 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.462975    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-run-systemd\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.463670 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463009    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-run-systemd\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.463670 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463024    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-run-ovn-kubernetes\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.463670 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463054    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e49068d1-9ae4-47b3-9a07-3d695a0a2402-system-cni-dir\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.463670 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463104    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-sysctl-d\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.463670 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463107    2571 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 01:50:31.463670 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463119    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-run-ovn-kubernetes\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.463670 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463143    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcpqh\" (UniqueName: \"kubernetes.io/projected/e49068d1-9ae4-47b3-9a07-3d695a0a2402-kube-api-access-jcpqh\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.463670 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463169    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e49068d1-9ae4-47b3-9a07-3d695a0a2402-system-cni-dir\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.463670 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463149    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-sysctl-conf\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.463670 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463209    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-conf-dir\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.463670 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463232    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-daemon-config\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.463670 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463276    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f0923bb-db25-4266-ad8d-2e1cfb7f602b-konnectivity-ca\") pod \"konnectivity-agent-vvzws\" (UID: \"1f0923bb-db25-4266-ad8d-2e1cfb7f602b\") " pod="kube-system/konnectivity-agent-vvzws"
Nov 03 01:50:31.463670 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463319    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-conf-dir\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.463670 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463320    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-etc-kubernetes\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.463670 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463353    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhg5z\" (UniqueName: \"kubernetes.io/projected/9260d15b-9772-4cb4-8588-8e82f89f39b8-kube-api-access-hhg5z\") pod \"node-ca-qg2kq\" (UID: \"9260d15b-9772-4cb4-8588-8e82f89f39b8\") " pod="openshift-image-registry/node-ca-qg2kq"
Nov 03 01:50:31.463670 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463362    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-etc-kubernetes\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.464572 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463399    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-tmp\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.464572 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463513    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-ovnkube-script-lib\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.464572 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463556    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b57cc991-d758-435f-9238-0dac97f10a93-iptables-alerter-script\") pod \"iptables-alerter-tp5hd\" (UID: \"b57cc991-d758-435f-9238-0dac97f10a93\") " pod="openshift-network-operator/iptables-alerter-tp5hd"
Nov 03 01:50:31.464572 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463662    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-var-lib-kubelet\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.464572 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463914    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-kubelet\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.464572 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463949    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-etc-openvswitch\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.464572 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463763    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-var-lib-kubelet\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.464572 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.463975    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-node-log\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.464572 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464000    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-cni-netd\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.464572 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464011    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-kubelet\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.464572 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464025    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-ovnkube-config\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.464572 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464050    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-node-log\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.464572 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464051    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq5lq\" (UniqueName: \"kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq\") pod \"network-check-target-mdr8p\" (UID: \"42a56b4b-283d-4e63-bb6a-59c5d5df0c2c\") " pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:50:31.464572 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464084    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs\") pod \"network-metrics-daemon-xv8zr\" (UID: \"c7a013c2-18a4-4372-adcb-8dfafa673f9c\") " pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:31.464572 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464112    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-sysconfig\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.464572 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464122    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-ovnkube-script-lib\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.464572 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464132    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-kubernetes\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.465399 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464174    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:31.465399 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464197    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltkx2\" (UniqueName: \"kubernetes.io/projected/c7a013c2-18a4-4372-adcb-8dfafa673f9c-kube-api-access-ltkx2\") pod \"network-metrics-daemon-xv8zr\" (UID: \"c7a013c2-18a4-4372-adcb-8dfafa673f9c\") " pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:31.465399 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464217    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8684a429-234f-4537-917e-5b49509bbdb5-hosts-file\") pod \"node-resolver-p8kgq\" (UID: \"8684a429-234f-4537-917e-5b49509bbdb5\") " pod="openshift-dns/node-resolver-p8kgq"
Nov 03 01:50:31.465399 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464240    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl96f\" (UniqueName: \"kubernetes.io/projected/b57cc991-d758-435f-9238-0dac97f10a93-kube-api-access-sl96f\") pod \"iptables-alerter-tp5hd\" (UID: \"b57cc991-d758-435f-9238-0dac97f10a93\") " pod="openshift-network-operator/iptables-alerter-tp5hd"
Nov 03 01:50:31.465399 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464264    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-socket-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.465399 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464284    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-lib-modules\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.465399 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464306    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-var-lib-openvswitch\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.465399 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464327    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-cni-bin\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.465399 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464350    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e49068d1-9ae4-47b3-9a07-3d695a0a2402-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.465399 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464372    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-hostroot\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.465399 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464387    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-cni-netd\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.465399 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464394    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9260d15b-9772-4cb4-8588-8e82f89f39b8-serviceca\") pod \"node-ca-qg2kq\" (UID: \"9260d15b-9772-4cb4-8588-8e82f89f39b8\") " pod="openshift-image-registry/node-ca-qg2kq"
Nov 03 01:50:31.465399 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464420    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-slash\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.465399 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464441    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-run-ovn\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.465399 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464460    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79fe02fa-99ce-4222-b569-6b323df47ab4-dbus\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:31.465399 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464459    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8684a429-234f-4537-917e-5b49509bbdb5-hosts-file\") pod \"node-resolver-p8kgq\" (UID: \"8684a429-234f-4537-917e-5b49509bbdb5\") " pod="openshift-dns/node-resolver-p8kgq"
Nov 03 01:50:31.465399 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464485    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-socket-dir-parent\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.466230 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464509    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-var-lib-cni-bin\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.466230 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464532    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6lhl\" (UniqueName: \"kubernetes.io/projected/8684a429-234f-4537-917e-5b49509bbdb5-kube-api-access-h6lhl\") pod \"node-resolver-p8kgq\" (UID: \"8684a429-234f-4537-917e-5b49509bbdb5\") " pod="openshift-dns/node-resolver-p8kgq"
Nov 03 01:50:31.466230 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464534    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b57cc991-d758-435f-9238-0dac97f10a93-iptables-alerter-script\") pod \"iptables-alerter-tp5hd\" (UID: \"b57cc991-d758-435f-9238-0dac97f10a93\") " pod="openshift-network-operator/iptables-alerter-tp5hd"
Nov 03 01:50:31.466230 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464557    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-registration-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.466230 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464579    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-modprobe-d\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.466230 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:31.464586    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:50:31.466230 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464601    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-host\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.466230 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464632    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2d5x\" (UniqueName: \"kubernetes.io/projected/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-kube-api-access-f2d5x\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.466230 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:31.464647    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs podName:c7a013c2-18a4-4372-adcb-8dfafa673f9c nodeName:}" failed. No retries permitted until 2026-11-03 01:50:31.964628227 +0000 UTC m=+2.252199024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs") pod "network-metrics-daemon-xv8zr" (UID: "c7a013c2-18a4-4372-adcb-8dfafa673f9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:50:31.466230 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464680    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e49068d1-9ae4-47b3-9a07-3d695a0a2402-cnibin\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.466230 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464701    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-slash\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.466230 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464711    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79fe02fa-99ce-4222-b569-6b323df47ab4-kubelet-config\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:31.466230 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464739    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b57cc991-d758-435f-9238-0dac97f10a93-host-slash\") pod \"iptables-alerter-tp5hd\" (UID: \"b57cc991-d758-435f-9238-0dac97f10a93\") " pod="openshift-network-operator/iptables-alerter-tp5hd"
Nov 03 01:50:31.466230 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464753    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-run-ovn\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.466230 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464765    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-run-openvswitch\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.466230 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464810    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8684a429-234f-4537-917e-5b49509bbdb5-tmp-dir\") pod \"node-resolver-p8kgq\" (UID: \"8684a429-234f-4537-917e-5b49509bbdb5\") " pod="openshift-dns/node-resolver-p8kgq"
Nov 03 01:50:31.466230 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464831    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79fe02fa-99ce-4222-b569-6b323df47ab4-dbus\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:31.467050 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464848    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.467050 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464509    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-etc-openvswitch\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.467050 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464885    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-ovnkube-config\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.467050 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464892    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-var-lib-kubelet\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.467050 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464889    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-socket-dir-parent\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.467050 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464915    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-sysconfig\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.467050 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464921    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-tuned\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.467050 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464924    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-var-lib-cni-bin\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.467050 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464949    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.467050 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464959    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b57cc991-d758-435f-9238-0dac97f10a93-host-slash\") pod \"iptables-alerter-tp5hd\" (UID: \"b57cc991-d758-435f-9238-0dac97f10a93\") " pod="openshift-network-operator/iptables-alerter-tp5hd"
Nov 03 01:50:31.467050 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464979    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-env-overrides\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.467050 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.464996    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e49068d1-9ae4-47b3-9a07-3d695a0a2402-cnibin\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.467050 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465003    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-kubernetes\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.467050 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465050    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79fe02fa-99ce-4222-b569-6b323df47ab4-kubelet-config\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:31.467050 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465080    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9260d15b-9772-4cb4-8588-8e82f89f39b8-serviceca\") pod \"node-ca-qg2kq\" (UID: \"9260d15b-9772-4cb4-8588-8e82f89f39b8\") " pod="openshift-image-registry/node-ca-qg2kq"
Nov 03 01:50:31.467050 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465089    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-var-lib-kubelet\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.467050 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:31.465165    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 01:50:31.467050 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465197    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-socket-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.467792 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:31.465222    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret podName:79fe02fa-99ce-4222-b569-6b323df47ab4 nodeName:}" failed. No retries permitted until 2026-11-03 01:50:31.965204891 +0000 UTC m=+2.252775685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret") pod "global-pull-secret-syncer-mxg52" (UID: "79fe02fa-99ce-4222-b569-6b323df47ab4") : object "kube-system"/"original-pull-secret" not registered
Nov 03 01:50:31.467792 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465240    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8684a429-234f-4537-917e-5b49509bbdb5-tmp-dir\") pod \"node-resolver-p8kgq\" (UID: \"8684a429-234f-4537-917e-5b49509bbdb5\") " pod="openshift-dns/node-resolver-p8kgq"
Nov 03 01:50:31.467792 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465258    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-registration-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.467792 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465293    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-lib-modules\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.467792 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465344    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-modprobe-d\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.467792 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465350    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-cni-bin\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.467792 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465353    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-var-lib-openvswitch\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.467792 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465392    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-hostroot\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.467792 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465006    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-cni-binary-copy\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.467792 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465457    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-run-openvswitch\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.467792 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465457    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.467792 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465481    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-run-netns\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.467792 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465511    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h48h\" (UniqueName: \"kubernetes.io/projected/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-kube-api-access-5h48h\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.467792 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465540    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e49068d1-9ae4-47b3-9a07-3d695a0a2402-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.467792 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465565    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9260d15b-9772-4cb4-8588-8e82f89f39b8-host\") pod \"node-ca-qg2kq\" (UID: \"9260d15b-9772-4cb4-8588-8e82f89f39b8\") " pod="openshift-image-registry/node-ca-qg2kq"
Nov 03 01:50:31.467792 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465589    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-device-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.467792 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465612    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-systemd\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.468556 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465637    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-sys\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.468556 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465661    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flcc2\" (UniqueName: \"kubernetes.io/projected/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-kube-api-access-flcc2\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.468556 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465686    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e49068d1-9ae4-47b3-9a07-3d695a0a2402-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.468556 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465702    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.468556 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465713    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-ovn-node-metrics-cert\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.468556 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465742    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e49068d1-9ae4-47b3-9a07-3d695a0a2402-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.468556 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465785    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-host\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.468556 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465798    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-run\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.468556 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465842    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-system-cni-dir\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.468556 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465847    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-run\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.468556 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465896    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-env-overrides\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.468556 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465906    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9260d15b-9772-4cb4-8588-8e82f89f39b8-host\") pod \"node-ca-qg2kq\" (UID: \"9260d15b-9772-4cb4-8588-8e82f89f39b8\") " pod="openshift-image-registry/node-ca-qg2kq"
Nov 03 01:50:31.468556 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465947    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-run-netns\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.468556 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.466020    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e49068d1-9ae4-47b3-9a07-3d695a0a2402-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.468556 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.466088    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-sys\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.468556 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.465511    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-cni-binary-copy\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.468556 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.466135    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-device-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.468556 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.466192    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-systemd\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.469170 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.466543    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-system-cni-dir\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.469170 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.467053    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e49068d1-9ae4-47b3-9a07-3d695a0a2402-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.469170 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.467828    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-tmp\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.469170 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.467885    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-tuned\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.469170 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.467921    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e49068d1-9ae4-47b3-9a07-3d695a0a2402-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.469170 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.467958    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e49068d1-9ae4-47b3-9a07-3d695a0a2402-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.469170 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.468336    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-ovn-node-metrics-cert\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.469170 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.468415    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f0923bb-db25-4266-ad8d-2e1cfb7f602b-agent-certs\") pod \"konnectivity-agent-vvzws\" (UID: \"1f0923bb-db25-4266-ad8d-2e1cfb7f602b\") " pod="kube-system/konnectivity-agent-vvzws"
Nov 03 01:50:31.523578 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:31.523557    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 01:50:31.523717 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:31.523585    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 01:50:31.523717 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:31.523599    2571 projected.go:196] Error preparing data for projected volume kube-api-access-hq5lq for pod openshift-network-diagnostics/network-check-target-mdr8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:50:31.523717 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:31.523686    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq podName:42a56b4b-283d-4e63-bb6a-59c5d5df0c2c nodeName:}" failed. No retries permitted until 2026-11-03 01:50:32.023666283 +0000 UTC m=+2.311237076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hq5lq" (UniqueName: "kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq") pod "network-check-target-mdr8p" (UID: "42a56b4b-283d-4e63-bb6a-59c5d5df0c2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:50:31.526179 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.525999    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhg5z\" (UniqueName: \"kubernetes.io/projected/9260d15b-9772-4cb4-8588-8e82f89f39b8-kube-api-access-hhg5z\") pod \"node-ca-qg2kq\" (UID: \"9260d15b-9772-4cb4-8588-8e82f89f39b8\") " pod="openshift-image-registry/node-ca-qg2kq"
Nov 03 01:50:31.526179 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.526088    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcpqh\" (UniqueName: \"kubernetes.io/projected/e49068d1-9ae4-47b3-9a07-3d695a0a2402-kube-api-access-jcpqh\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.527517 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.526422    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqqkf\" (UniqueName: \"kubernetes.io/projected/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-kube-api-access-qqqkf\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.527517 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.527417    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl96f\" (UniqueName: \"kubernetes.io/projected/b57cc991-d758-435f-9238-0dac97f10a93-kube-api-access-sl96f\") pod \"iptables-alerter-tp5hd\" (UID: \"b57cc991-d758-435f-9238-0dac97f10a93\") " pod="openshift-network-operator/iptables-alerter-tp5hd"
Nov 03 01:50:31.527517 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.527421    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h48h\" (UniqueName: \"kubernetes.io/projected/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-kube-api-access-5h48h\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.527966 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.527941    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flcc2\" (UniqueName: \"kubernetes.io/projected/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-kube-api-access-flcc2\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.528110 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.528094    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltkx2\" (UniqueName: \"kubernetes.io/projected/c7a013c2-18a4-4372-adcb-8dfafa673f9c-kube-api-access-ltkx2\") pod \"network-metrics-daemon-xv8zr\" (UID: \"c7a013c2-18a4-4372-adcb-8dfafa673f9c\") " pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:31.528161 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.528130    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2d5x\" (UniqueName: \"kubernetes.io/projected/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-kube-api-access-f2d5x\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.528281 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.528266    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6lhl\" (UniqueName: \"kubernetes.io/projected/8684a429-234f-4537-917e-5b49509bbdb5-kube-api-access-h6lhl\") pod \"node-resolver-p8kgq\" (UID: \"8684a429-234f-4537-917e-5b49509bbdb5\") " pod="openshift-dns/node-resolver-p8kgq"
Nov 03 01:50:31.658527 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.658492    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:50:31.662517 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.662495    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:50:31.700589 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.700570    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:31.704517 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.704496    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9kq9k"
Nov 03 01:50:31.727956 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.727934    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:50:31.732890 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.732859    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vvzws"
Nov 03 01:50:31.751743 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.751723    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p8kgq"
Nov 03 01:50:31.772305 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.772283    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tp5hd"
Nov 03 01:50:31.782035 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.781989    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qg2kq"
Nov 03 01:50:31.970082 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.970044    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs\") pod \"network-metrics-daemon-xv8zr\" (UID: \"c7a013c2-18a4-4372-adcb-8dfafa673f9c\") " pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:31.970082 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:31.970077    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:31.970409 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:31.970188    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 01:50:31.970409 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:31.970190    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:50:31.970409 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:31.970258    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs podName:c7a013c2-18a4-4372-adcb-8dfafa673f9c nodeName:}" failed. No retries permitted until 2026-11-03 01:50:32.970230169 +0000 UTC m=+3.257800976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs") pod "network-metrics-daemon-xv8zr" (UID: "c7a013c2-18a4-4372-adcb-8dfafa673f9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:50:31.970409 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:31.970273    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret podName:79fe02fa-99ce-4222-b569-6b323df47ab4 nodeName:}" failed. No retries permitted until 2026-11-03 01:50:32.970266191 +0000 UTC m=+3.257836964 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret") pod "global-pull-secret-syncer-mxg52" (UID: "79fe02fa-99ce-4222-b569-6b323df47ab4") : object "kube-system"/"original-pull-secret" not registered
Nov 03 01:50:32.070988 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:32.070898    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq5lq\" (UniqueName: \"kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq\") pod \"network-check-target-mdr8p\" (UID: \"42a56b4b-283d-4e63-bb6a-59c5d5df0c2c\") " pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:50:32.071388 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:32.071055    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 01:50:32.071388 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:32.071078    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 01:50:32.071388 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:32.071090    2571 projected.go:196] Error preparing data for projected volume kube-api-access-hq5lq for pod openshift-network-diagnostics/network-check-target-mdr8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:50:32.071388 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:32.071140    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq podName:42a56b4b-283d-4e63-bb6a-59c5d5df0c2c nodeName:}" failed. No retries permitted until 2026-11-03 01:50:33.071124246 +0000 UTC m=+3.358695020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hq5lq" (UniqueName: "kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq") pod "network-check-target-mdr8p" (UID: "42a56b4b-283d-4e63-bb6a-59c5d5df0c2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:50:32.091316 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:32.091283    2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 01:50:32.092578 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:32.092540    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37711de5beea2605bd7d0ac9a465c416.slice/crio-fcd57864758c6e788a88dea6c88fb474427145eb04c28fd86dfce47b14f6d3b0 WatchSource:0}: Error finding container fcd57864758c6e788a88dea6c88fb474427145eb04c28fd86dfce47b14f6d3b0: Status 404 returned error can't find the container with id fcd57864758c6e788a88dea6c88fb474427145eb04c28fd86dfce47b14f6d3b0
Nov 03 01:50:32.097629 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:32.097606    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6076c2cc730e7ded3ca7127f0bb3e92b.slice/crio-8a2b0a911b8a92222130cb200c593379e9a7975d1527e41c6b12a12dae6af66e WatchSource:0}: Error finding container 8a2b0a911b8a92222130cb200c593379e9a7975d1527e41c6b12a12dae6af66e: Status 404 returned error can't find the container with id 8a2b0a911b8a92222130cb200c593379e9a7975d1527e41c6b12a12dae6af66e
Nov 03 01:50:32.099210 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:32.099193    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:50:32.174635 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:32.174605    2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 01:50:32.249114 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:32.249060    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal" event={"ID":"37711de5beea2605bd7d0ac9a465c416","Type":"ContainerStarted","Data":"fcd57864758c6e788a88dea6c88fb474427145eb04c28fd86dfce47b14f6d3b0"}
Nov 03 01:50:32.250218 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:32.250197    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal" event={"ID":"6076c2cc730e7ded3ca7127f0bb3e92b","Type":"ContainerStarted","Data":"8a2b0a911b8a92222130cb200c593379e9a7975d1527e41c6b12a12dae6af66e"}
Nov 03 01:50:32.323739 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:32.323650    2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 01:50:32.440669 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:32.440591    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 01:45:31 +0000 UTC" deadline="2028-04-03 08:43:10.349107232 +0000 UTC"
Nov 03 01:50:32.440669 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:32.440658    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12414h52m37.908451826s"
Nov 03 01:50:32.614880 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:32.614846    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ba28bb_4bab_4c0d_aa00_8b3bce675f23.slice/crio-7c4527b6974ef6c6920a016d149e89500669f178085b5e49dbfd029f2617c0fa WatchSource:0}: Error finding container 7c4527b6974ef6c6920a016d149e89500669f178085b5e49dbfd029f2617c0fa: Status 404 returned error can't find the container with id 7c4527b6974ef6c6920a016d149e89500669f178085b5e49dbfd029f2617c0fa
Nov 03 01:50:32.637617 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:32.637587    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30f499a_7d25_4ea7_aa1e_357ccbf0485f.slice/crio-c5ccfb7e0ba4c33f7f2ac59a18907866c0bc8256d609daf39884235e007e6176 WatchSource:0}: Error finding container c5ccfb7e0ba4c33f7f2ac59a18907866c0bc8256d609daf39884235e007e6176: Status 404 returned error can't find the container with id c5ccfb7e0ba4c33f7f2ac59a18907866c0bc8256d609daf39884235e007e6176
Nov 03 01:50:32.679737 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:32.679682    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8684a429_234f_4537_917e_5b49509bbdb5.slice/crio-a79a11b92c5252da0a40c17154db0cf7ca937cde26f1e2192e5143f2bd2c2b6c WatchSource:0}: Error finding container a79a11b92c5252da0a40c17154db0cf7ca937cde26f1e2192e5143f2bd2c2b6c: Status 404 returned error can't find the container with id a79a11b92c5252da0a40c17154db0cf7ca937cde26f1e2192e5143f2bd2c2b6c
Nov 03 01:50:32.680610 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:32.680577    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7a1afb_11cc_4aa4_bae6_68ce495ebca8.slice/crio-ba70ce368c30cd7c278ff369764b964bbb9d236b809ad498811a47824475f279 WatchSource:0}: Error finding container ba70ce368c30cd7c278ff369764b964bbb9d236b809ad498811a47824475f279: Status 404 returned error can't find the container with id ba70ce368c30cd7c278ff369764b964bbb9d236b809ad498811a47824475f279
Nov 03 01:50:32.713105 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:32.713076    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1ca62b_2da2_47a9_8e2a_34f2dc75a431.slice/crio-01c22ba375d44a764909a8b1a538cd58366f9f8a673bd396b940d0abf5084a78 WatchSource:0}: Error finding container 01c22ba375d44a764909a8b1a538cd58366f9f8a673bd396b940d0abf5084a78: Status 404 returned error can't find the container with id 01c22ba375d44a764909a8b1a538cd58366f9f8a673bd396b940d0abf5084a78
Nov 03 01:50:32.724223 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:32.724199    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49068d1_9ae4_47b3_9a07_3d695a0a2402.slice/crio-6d0ceec34113f5a8a9eb64e5c5e2899b435a56fa1464f3bb1c6292ddbff3a336 WatchSource:0}: Error finding container 6d0ceec34113f5a8a9eb64e5c5e2899b435a56fa1464f3bb1c6292ddbff3a336: Status 404 returned error can't find the container with id 6d0ceec34113f5a8a9eb64e5c5e2899b435a56fa1464f3bb1c6292ddbff3a336
Nov 03 01:50:32.742597 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:32.742561    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0923bb_db25_4266_ad8d_2e1cfb7f602b.slice/crio-c1f69404f3c50f789279e5a3757d9a8b1b4787b5e39d06df736c090657cfed25 WatchSource:0}: Error finding container c1f69404f3c50f789279e5a3757d9a8b1b4787b5e39d06df736c090657cfed25: Status 404 returned error can't find the container with id c1f69404f3c50f789279e5a3757d9a8b1b4787b5e39d06df736c090657cfed25
Nov 03 01:50:32.798291 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:32.798261    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9260d15b_9772_4cb4_8588_8e82f89f39b8.slice/crio-80c2f3b680f02615b275e52c25d8dca0b759e27c479f7ef19bc486e6d75cd70b WatchSource:0}: Error finding container 80c2f3b680f02615b275e52c25d8dca0b759e27c479f7ef19bc486e6d75cd70b: Status 404 returned error can't find the container with id 80c2f3b680f02615b275e52c25d8dca0b759e27c479f7ef19bc486e6d75cd70b
Nov 03 01:50:32.808433 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:50:32.808409    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57cc991_d758_435f_9238_0dac97f10a93.slice/crio-b02ad7dea43ede949cc8a9433f233c2a12cb95d0768b423eb90ce3478e7c1b85 WatchSource:0}: Error finding container b02ad7dea43ede949cc8a9433f233c2a12cb95d0768b423eb90ce3478e7c1b85: Status 404 returned error can't find the container with id b02ad7dea43ede949cc8a9433f233c2a12cb95d0768b423eb90ce3478e7c1b85
Nov 03 01:50:32.978535 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:32.978427    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs\") pod \"network-metrics-daemon-xv8zr\" (UID: \"c7a013c2-18a4-4372-adcb-8dfafa673f9c\") " pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:32.978535 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:32.978464    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:32.978808 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:32.978595    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 01:50:32.978808 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:32.978620    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:50:32.978808 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:32.978649    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret podName:79fe02fa-99ce-4222-b569-6b323df47ab4 nodeName:}" failed. No retries permitted until 2026-11-03 01:50:34.978634224 +0000 UTC m=+5.266204997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret") pod "global-pull-secret-syncer-mxg52" (UID: "79fe02fa-99ce-4222-b569-6b323df47ab4") : object "kube-system"/"original-pull-secret" not registered
Nov 03 01:50:32.978808 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:32.978689    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs podName:c7a013c2-18a4-4372-adcb-8dfafa673f9c nodeName:}" failed. No retries permitted until 2026-11-03 01:50:34.978668765 +0000 UTC m=+5.266239551 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs") pod "network-metrics-daemon-xv8zr" (UID: "c7a013c2-18a4-4372-adcb-8dfafa673f9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:50:33.079771 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:33.079731    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq5lq\" (UniqueName: \"kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq\") pod \"network-check-target-mdr8p\" (UID: \"42a56b4b-283d-4e63-bb6a-59c5d5df0c2c\") " pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:50:33.080346 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:33.079895    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 01:50:33.080346 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:33.079917    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 01:50:33.080346 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:33.079928    2571 projected.go:196] Error preparing data for projected volume kube-api-access-hq5lq for pod openshift-network-diagnostics/network-check-target-mdr8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:50:33.080346 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:33.079980    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq podName:42a56b4b-283d-4e63-bb6a-59c5d5df0c2c nodeName:}" failed. No retries permitted until 2026-11-03 01:50:35.079966865 +0000 UTC m=+5.367537639 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hq5lq" (UniqueName: "kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq") pod "network-check-target-mdr8p" (UID: "42a56b4b-283d-4e63-bb6a-59c5d5df0c2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:50:33.241654 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:33.241571    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:33.241654 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:33.241598    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:33.241654 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:33.241612    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:50:33.241959 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:33.241695    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv8zr" podUID="c7a013c2-18a4-4372-adcb-8dfafa673f9c"
Nov 03 01:50:33.241959 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:33.241797    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mxg52" podUID="79fe02fa-99ce-4222-b569-6b323df47ab4"
Nov 03 01:50:33.241959 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:33.241895    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdr8p" podUID="42a56b4b-283d-4e63-bb6a-59c5d5df0c2c"
Nov 03 01:50:33.253315 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:33.253284    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vvzws" event={"ID":"1f0923bb-db25-4266-ad8d-2e1cfb7f602b","Type":"ContainerStarted","Data":"c1f69404f3c50f789279e5a3757d9a8b1b4787b5e39d06df736c090657cfed25"}
Nov 03 01:50:33.254462 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:33.254442    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-25wbg" event={"ID":"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431","Type":"ContainerStarted","Data":"01c22ba375d44a764909a8b1a538cd58366f9f8a673bd396b940d0abf5084a78"}
Nov 03 01:50:33.255635 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:33.255617    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p8kgq" event={"ID":"8684a429-234f-4537-917e-5b49509bbdb5","Type":"ContainerStarted","Data":"a79a11b92c5252da0a40c17154db0cf7ca937cde26f1e2192e5143f2bd2c2b6c"}
Nov 03 01:50:33.256773 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:33.256752    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tp5hd" event={"ID":"b57cc991-d758-435f-9238-0dac97f10a93","Type":"ContainerStarted","Data":"b02ad7dea43ede949cc8a9433f233c2a12cb95d0768b423eb90ce3478e7c1b85"}
Nov 03 01:50:33.257694 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:33.257676    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qg2kq" event={"ID":"9260d15b-9772-4cb4-8588-8e82f89f39b8","Type":"ContainerStarted","Data":"80c2f3b680f02615b275e52c25d8dca0b759e27c479f7ef19bc486e6d75cd70b"}
Nov 03 01:50:33.258746 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:33.258724    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7qsp" event={"ID":"e49068d1-9ae4-47b3-9a07-3d695a0a2402","Type":"ContainerStarted","Data":"6d0ceec34113f5a8a9eb64e5c5e2899b435a56fa1464f3bb1c6292ddbff3a336"}
Nov 03 01:50:33.259803 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:33.259785    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9kq9k" event={"ID":"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8","Type":"ContainerStarted","Data":"ba70ce368c30cd7c278ff369764b964bbb9d236b809ad498811a47824475f279"}
Nov 03 01:50:33.260952 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:33.260924    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs" event={"ID":"f30f499a-7d25-4ea7-aa1e-357ccbf0485f","Type":"ContainerStarted","Data":"c5ccfb7e0ba4c33f7f2ac59a18907866c0bc8256d609daf39884235e007e6176"}
Nov 03 01:50:33.261965 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:33.261946    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jdk59" event={"ID":"54ba28bb-4bab-4c0d-aa00-8b3bce675f23","Type":"ContainerStarted","Data":"7c4527b6974ef6c6920a016d149e89500669f178085b5e49dbfd029f2617c0fa"}
Nov 03 01:50:33.441178 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:33.441124    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 01:45:31 +0000 UTC" deadline="2028-07-01 20:19:38.992895456 +0000 UTC"
Nov 03 01:50:33.441178 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:33.441166    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14562h29m5.551734118s"
Nov 03 01:50:34.997682 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:34.997588    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs\") pod \"network-metrics-daemon-xv8zr\" (UID: \"c7a013c2-18a4-4372-adcb-8dfafa673f9c\") " pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:34.997682 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:34.997644    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:34.998524 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:34.997793    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 01:50:34.998524 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:34.997861    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret podName:79fe02fa-99ce-4222-b569-6b323df47ab4 nodeName:}" failed. No retries permitted until 2026-11-03 01:50:38.997840993 +0000 UTC m=+9.285411793 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret") pod "global-pull-secret-syncer-mxg52" (UID: "79fe02fa-99ce-4222-b569-6b323df47ab4") : object "kube-system"/"original-pull-secret" not registered
Nov 03 01:50:34.998524 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:34.997934    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:50:34.998524 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:34.997970    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs podName:c7a013c2-18a4-4372-adcb-8dfafa673f9c nodeName:}" failed. No retries permitted until 2026-11-03 01:50:38.997958073 +0000 UTC m=+9.285528850 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs") pod "network-metrics-daemon-xv8zr" (UID: "c7a013c2-18a4-4372-adcb-8dfafa673f9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:50:35.098159 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:35.098120    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq5lq\" (UniqueName: \"kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq\") pod \"network-check-target-mdr8p\" (UID: \"42a56b4b-283d-4e63-bb6a-59c5d5df0c2c\") " pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:50:35.098372 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:35.098323    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 01:50:35.098372 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:35.098351    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 01:50:35.098372 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:35.098364    2571 projected.go:196] Error preparing data for projected volume kube-api-access-hq5lq for pod openshift-network-diagnostics/network-check-target-mdr8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:50:35.098555 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:35.098437    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq podName:42a56b4b-283d-4e63-bb6a-59c5d5df0c2c nodeName:}" failed. No retries permitted until 2026-11-03 01:50:39.0984163 +0000 UTC m=+9.385987102 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hq5lq" (UniqueName: "kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq") pod "network-check-target-mdr8p" (UID: "42a56b4b-283d-4e63-bb6a-59c5d5df0c2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:50:35.242143 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:35.242110    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:35.242339 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:35.242242    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv8zr" podUID="c7a013c2-18a4-4372-adcb-8dfafa673f9c"
Nov 03 01:50:35.242339 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:35.242335    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:35.242482 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:35.242421    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mxg52" podUID="79fe02fa-99ce-4222-b569-6b323df47ab4"
Nov 03 01:50:35.242541 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:35.242484    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:50:35.242597 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:35.242559    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdr8p" podUID="42a56b4b-283d-4e63-bb6a-59c5d5df0c2c"
Nov 03 01:50:37.241866 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:37.241199    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:37.241866 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:37.241606    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:37.241866 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:37.241694    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mxg52" podUID="79fe02fa-99ce-4222-b569-6b323df47ab4"
Nov 03 01:50:37.241866 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:37.241594    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv8zr" podUID="c7a013c2-18a4-4372-adcb-8dfafa673f9c"
Nov 03 01:50:37.241866 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:37.241749    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:50:37.241866 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:37.241835    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdr8p" podUID="42a56b4b-283d-4e63-bb6a-59c5d5df0c2c"
Nov 03 01:50:37.287456 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:37.286489    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal" event={"ID":"37711de5beea2605bd7d0ac9a465c416","Type":"ContainerStarted","Data":"9973f3a32c179e2ebb266c3b38926cebea8e27de5aa01f7e62f64e7999d4f934"}
Nov 03 01:50:38.308399 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:38.307713    2571 generic.go:356] "Generic (PLEG): container finished" podID="e49068d1-9ae4-47b3-9a07-3d695a0a2402" containerID="a3869b735a7d1363920c3adf674317a711fc4c5836981f79765233b6631d3562" exitCode=0
Nov 03 01:50:38.308399 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:38.307846    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7qsp" event={"ID":"e49068d1-9ae4-47b3-9a07-3d695a0a2402","Type":"ContainerDied","Data":"a3869b735a7d1363920c3adf674317a711fc4c5836981f79765233b6631d3562"}
Nov 03 01:50:38.317328 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:38.317292    2571 generic.go:356] "Generic (PLEG): container finished" podID="6076c2cc730e7ded3ca7127f0bb3e92b" containerID="dae4a6cea5adfe867282e761d33fd8007491853908768ea609e1e2c0915b0e1d" exitCode=0
Nov 03 01:50:38.318266 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:38.318239    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal" event={"ID":"6076c2cc730e7ded3ca7127f0bb3e92b","Type":"ContainerDied","Data":"dae4a6cea5adfe867282e761d33fd8007491853908768ea609e1e2c0915b0e1d"}
Nov 03 01:50:38.535409 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:38.535363    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal" podStartSLOduration=7.535348546 podStartE2EDuration="7.535348546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:37.410934909 +0000 UTC m=+7.698505708" watchObservedRunningTime="2026-11-03 01:50:38.535348546 +0000 UTC m=+8.822919405"
Nov 03 01:50:39.035426 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:39.034002    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs\") pod \"network-metrics-daemon-xv8zr\" (UID: \"c7a013c2-18a4-4372-adcb-8dfafa673f9c\") " pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:39.035426 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:39.034050    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:39.035426 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:39.034217    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 01:50:39.035426 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:39.034279    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret podName:79fe02fa-99ce-4222-b569-6b323df47ab4 nodeName:}" failed. No retries permitted until 2026-11-03 01:50:47.03425911 +0000 UTC m=+17.321829887 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret") pod "global-pull-secret-syncer-mxg52" (UID: "79fe02fa-99ce-4222-b569-6b323df47ab4") : object "kube-system"/"original-pull-secret" not registered
Nov 03 01:50:39.035426 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:39.034346    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:50:39.035426 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:39.034379    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs podName:c7a013c2-18a4-4372-adcb-8dfafa673f9c nodeName:}" failed. No retries permitted until 2026-11-03 01:50:47.034366791 +0000 UTC m=+17.321937580 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs") pod "network-metrics-daemon-xv8zr" (UID: "c7a013c2-18a4-4372-adcb-8dfafa673f9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:50:39.134868 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:39.134839    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq5lq\" (UniqueName: \"kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq\") pod \"network-check-target-mdr8p\" (UID: \"42a56b4b-283d-4e63-bb6a-59c5d5df0c2c\") " pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:50:39.135056 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:39.135035    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 01:50:39.135136 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:39.135063    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 01:50:39.135136 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:39.135076    2571 projected.go:196] Error preparing data for projected volume kube-api-access-hq5lq for pod openshift-network-diagnostics/network-check-target-mdr8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:50:39.135136 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:39.135127    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq podName:42a56b4b-283d-4e63-bb6a-59c5d5df0c2c nodeName:}" failed. No retries permitted until 2026-11-03 01:50:47.135109576 +0000 UTC m=+17.422680363 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hq5lq" (UniqueName: "kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq") pod "network-check-target-mdr8p" (UID: "42a56b4b-283d-4e63-bb6a-59c5d5df0c2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:50:39.241932 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:39.241902    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:39.242113 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:39.242044    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv8zr" podUID="c7a013c2-18a4-4372-adcb-8dfafa673f9c"
Nov 03 01:50:39.242166 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:39.242125    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:39.242221 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:39.242204    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mxg52" podUID="79fe02fa-99ce-4222-b569-6b323df47ab4"
Nov 03 01:50:39.242276 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:39.242263    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:50:39.242352 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:39.242331    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdr8p" podUID="42a56b4b-283d-4e63-bb6a-59c5d5df0c2c"
Nov 03 01:50:39.324920 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:39.324880    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qg2kq" event={"ID":"9260d15b-9772-4cb4-8588-8e82f89f39b8","Type":"ContainerStarted","Data":"b4e0e46bd422516a1cc371d56e8e4e9eba4bc6282523a0f9bacb66ebe4ee43f2"}
Nov 03 01:50:39.328174 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:39.328147    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal" event={"ID":"6076c2cc730e7ded3ca7127f0bb3e92b","Type":"ContainerStarted","Data":"5d829ef3c1903ffa6e014a1b0b39feb49170a1a6a528261a80b5c984af7b193a"}
Nov 03 01:50:39.330968 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:39.330945    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vvzws" event={"ID":"1f0923bb-db25-4266-ad8d-2e1cfb7f602b","Type":"ContainerStarted","Data":"822c054614462482e19f3aeb410f5abfefdcb2f12610d34c34b272bed60e911a"}
Nov 03 01:50:39.439107 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:39.439014    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qg2kq" podStartSLOduration=2.285454645 podStartE2EDuration="8.438996388s" totalImagesPullingTime="6.153541743s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:39.438001709 +0000 UTC m=+9.725572508" watchObservedRunningTime="2026-11-03 01:50:39.438996388 +0000 UTC m=+9.726567185"
Nov 03 01:50:39.666538 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:39.666478    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal" podStartSLOduration=8.666460353 podStartE2EDuration="8.666460353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:39.551956298 +0000 UTC m=+9.839527093" watchObservedRunningTime="2026-11-03 01:50:39.666460353 +0000 UTC m=+9.954031151"
Nov 03 01:50:40.337164 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:40.337132    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs" event={"ID":"f30f499a-7d25-4ea7-aa1e-357ccbf0485f","Type":"ContainerStarted","Data":"f5d42040645dcf3c9662e203e6f9db4d10762e38aaf1052e05ce1e0fee2be2e7"}
Nov 03 01:50:41.242006 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:41.241965    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:41.242172 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:41.242128    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv8zr" podUID="c7a013c2-18a4-4372-adcb-8dfafa673f9c"
Nov 03 01:50:41.242230 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:41.242214    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:41.242366 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:41.242295    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mxg52" podUID="79fe02fa-99ce-4222-b569-6b323df47ab4"
Nov 03 01:50:41.242494 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:41.242368    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:50:41.242494 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:41.242436    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdr8p" podUID="42a56b4b-283d-4e63-bb6a-59c5d5df0c2c"
Nov 03 01:50:41.340906 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:41.340821    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9kq9k" event={"ID":"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8","Type":"ContainerStarted","Data":"90de7bf4bd97e4e4a8a9242d579b60fce80892131f6ba9415918525eadeb33be"}
Nov 03 01:50:41.456724 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:41.456658    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vvzws" podStartSLOduration=5.09991593 podStartE2EDuration="10.456645199s" totalImagesPullingTime="5.356729269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:39.667136563 +0000 UTC m=+9.954707361" watchObservedRunningTime="2026-11-03 01:50:41.456645199 +0000 UTC m=+11.744215992"
Nov 03 01:50:42.119915 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:42.119879    2571 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 01:50:42.132712 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:42.132385    2571 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T01:50:42.11989932Z","UUID":"4a194153-6a38-4c73-a16e-4b8954eef91b","Handler":null,"Name":"","Endpoint":""}
Nov 03 01:50:42.135473 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:42.135454    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 01:50:42.135558 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:42.135482    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 01:50:42.345446 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:42.345407    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs" event={"ID":"f30f499a-7d25-4ea7-aa1e-357ccbf0485f","Type":"ContainerStarted","Data":"277e55c71069955d06ae9e919f6e670933123f9ed6b481a5cf88d9e6a0055cc9"}
Nov 03 01:50:42.347224 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:42.347194    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p8kgq" event={"ID":"8684a429-234f-4537-917e-5b49509bbdb5","Type":"ContainerStarted","Data":"1b3350f58a168289dae604d8b7080452679a463cb33ab7fee386f5f2500ce9c6"}
Nov 03 01:50:42.463735 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:42.463621    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-9kq9k" podStartSLOduration=4.132005913 podStartE2EDuration="12.463602915s" totalImagesPullingTime="8.331597002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:41.456967917 +0000 UTC m=+11.744538762" watchObservedRunningTime="2026-11-03 01:50:42.463602915 +0000 UTC m=+12.751173713"
Nov 03 01:50:43.034210 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:43.034163    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vvzws"
Nov 03 01:50:43.241872 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:43.241846    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:50:43.241872 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:43.241864    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:43.242054 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:43.241972    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdr8p" podUID="42a56b4b-283d-4e63-bb6a-59c5d5df0c2c"
Nov 03 01:50:43.242089 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:43.242056    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mxg52" podUID="79fe02fa-99ce-4222-b569-6b323df47ab4"
Nov 03 01:50:43.242121 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:43.242107    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:43.242201 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:43.242181    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv8zr" podUID="c7a013c2-18a4-4372-adcb-8dfafa673f9c"
Nov 03 01:50:43.246940 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:43.246896    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vvzws"
Nov 03 01:50:43.247637 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:43.247615    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vvzws"
Nov 03 01:50:43.352065 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:43.351983    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tp5hd" event={"ID":"b57cc991-d758-435f-9238-0dac97f10a93","Type":"ContainerStarted","Data":"72c3a791beb3a24e6ea4a8fbbd50be0fb72ab4375391a480cbc3992a641f654b"}
Nov 03 01:50:43.354854 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:43.354826    2571 generic.go:356] "Generic (PLEG): container finished" podID="e49068d1-9ae4-47b3-9a07-3d695a0a2402" containerID="3e9a85060a97f5d0567360cd0401796581ebfa967dc634ce8662739a801c611a" exitCode=0
Nov 03 01:50:43.355028 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:43.354998    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7qsp" event={"ID":"e49068d1-9ae4-47b3-9a07-3d695a0a2402","Type":"ContainerDied","Data":"3e9a85060a97f5d0567360cd0401796581ebfa967dc634ce8662739a801c611a"}
Nov 03 01:50:43.356296 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:43.355925    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vvzws"
Nov 03 01:50:43.361133 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:43.361089    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-p8kgq" podStartSLOduration=3.142290736 podStartE2EDuration="12.361073936s" totalImagesPullingTime="9.2187832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:42.464434265 +0000 UTC m=+12.752005060" watchObservedRunningTime="2026-11-03 01:50:43.361073936 +0000 UTC m=+13.648644732"
Nov 03 01:50:43.585521 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:43.585474    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tp5hd" podStartSLOduration=3.946795357 podStartE2EDuration="12.585458134s" totalImagesPullingTime="8.638662777s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:43.584537035 +0000 UTC m=+13.872107831" watchObservedRunningTime="2026-11-03 01:50:43.585458134 +0000 UTC m=+13.873028928"
Nov 03 01:50:44.358239 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:44.358207    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs" event={"ID":"f30f499a-7d25-4ea7-aa1e-357ccbf0485f","Type":"ContainerStarted","Data":"37d982ced37c6d9e45d25d8a62b423d4cf43ef7470f27ec1685dcf3829afdce3"}
Nov 03 01:50:44.359741 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:44.359718    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-25wbg" event={"ID":"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431","Type":"ContainerStarted","Data":"4d9422f5b5a5786a77336b73ed307840e22769dc3b358ddbaab162a051dc78d7"}
Nov 03 01:50:44.472627 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:44.472587    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs" podStartSLOduration=3.643832523 podStartE2EDuration="14.472574281s" totalImagesPullingTime="10.828741758s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:44.472406776 +0000 UTC m=+14.759977570" watchObservedRunningTime="2026-11-03 01:50:44.472574281 +0000 UTC m=+14.760145077"
Nov 03 01:50:45.241372 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:45.241305    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:45.241372 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:45.241339    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:45.241546 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:45.241422    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:50:45.241546 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:45.241434    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv8zr" podUID="c7a013c2-18a4-4372-adcb-8dfafa673f9c"
Nov 03 01:50:45.241546 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:45.241490    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mxg52" podUID="79fe02fa-99ce-4222-b569-6b323df47ab4"
Nov 03 01:50:45.241672 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:45.241580    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdr8p" podUID="42a56b4b-283d-4e63-bb6a-59c5d5df0c2c"
Nov 03 01:50:45.364100 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:45.364065    2571 generic.go:356] "Generic (PLEG): container finished" podID="e49068d1-9ae4-47b3-9a07-3d695a0a2402" containerID="926b1dbf62a0a1fbe202cfb6da2b50d1d205e96072e6d0001cf1cc6807eec7fa" exitCode=0
Nov 03 01:50:45.364804 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:45.364147    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7qsp" event={"ID":"e49068d1-9ae4-47b3-9a07-3d695a0a2402","Type":"ContainerDied","Data":"926b1dbf62a0a1fbe202cfb6da2b50d1d205e96072e6d0001cf1cc6807eec7fa"}
Nov 03 01:50:45.481687 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:45.481636    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-25wbg" podStartSLOduration=4.622253655 podStartE2EDuration="15.481623453s" totalImagesPullingTime="10.859369798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:44.589209185 +0000 UTC m=+14.876779973" watchObservedRunningTime="2026-11-03 01:50:45.481623453 +0000 UTC m=+15.769194246"
Nov 03 01:50:47.094056 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:47.094024    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs\") pod \"network-metrics-daemon-xv8zr\" (UID: \"c7a013c2-18a4-4372-adcb-8dfafa673f9c\") " pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:47.094597 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:47.094064    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:47.094597 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:47.094168    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:50:47.094597 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:47.094224    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs podName:c7a013c2-18a4-4372-adcb-8dfafa673f9c nodeName:}" failed. No retries permitted until 2026-11-03 01:51:03.094210756 +0000 UTC m=+33.381781529 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs") pod "network-metrics-daemon-xv8zr" (UID: "c7a013c2-18a4-4372-adcb-8dfafa673f9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:50:47.094597 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:47.094168    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 01:50:47.094597 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:47.094317    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret podName:79fe02fa-99ce-4222-b569-6b323df47ab4 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:03.094302088 +0000 UTC m=+33.381872876 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret") pod "global-pull-secret-syncer-mxg52" (UID: "79fe02fa-99ce-4222-b569-6b323df47ab4") : object "kube-system"/"original-pull-secret" not registered
Nov 03 01:50:47.194879 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:47.194849    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq5lq\" (UniqueName: \"kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq\") pod \"network-check-target-mdr8p\" (UID: \"42a56b4b-283d-4e63-bb6a-59c5d5df0c2c\") " pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:50:47.195056 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:47.195042    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 01:50:47.195129 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:47.195063    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 01:50:47.195129 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:47.195076    2571 projected.go:196] Error preparing data for projected volume kube-api-access-hq5lq for pod openshift-network-diagnostics/network-check-target-mdr8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:50:47.195212 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:47.195133    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq podName:42a56b4b-283d-4e63-bb6a-59c5d5df0c2c nodeName:}" failed. No retries permitted until 2026-11-03 01:51:03.195116542 +0000 UTC m=+33.482687320 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hq5lq" (UniqueName: "kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq") pod "network-check-target-mdr8p" (UID: "42a56b4b-283d-4e63-bb6a-59c5d5df0c2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:50:47.241731 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:47.241689    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:47.241731 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:47.241718    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:50:47.241889 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:47.241821    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:47.241889 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:47.241829    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv8zr" podUID="c7a013c2-18a4-4372-adcb-8dfafa673f9c"
Nov 03 01:50:47.241982 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:47.241890    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdr8p" podUID="42a56b4b-283d-4e63-bb6a-59c5d5df0c2c"
Nov 03 01:50:47.242027 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:47.241974    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mxg52" podUID="79fe02fa-99ce-4222-b569-6b323df47ab4"
Nov 03 01:50:48.373635 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:48.373409    2571 generic.go:356] "Generic (PLEG): container finished" podID="e49068d1-9ae4-47b3-9a07-3d695a0a2402" containerID="a681afd79e83f58433cfd20760f2c818d47c622035cd44a992bf3ccc6d9c1a9d" exitCode=0
Nov 03 01:50:48.373635 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:48.373488    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7qsp" event={"ID":"e49068d1-9ae4-47b3-9a07-3d695a0a2402","Type":"ContainerDied","Data":"a681afd79e83f58433cfd20760f2c818d47c622035cd44a992bf3ccc6d9c1a9d"}
Nov 03 01:50:49.241911 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:49.241832    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:50:49.241911 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:49.241901    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:49.242095 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:49.241983    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdr8p" podUID="42a56b4b-283d-4e63-bb6a-59c5d5df0c2c"
Nov 03 01:50:49.242095 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:49.241995    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:49.242095 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:49.242069    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mxg52" podUID="79fe02fa-99ce-4222-b569-6b323df47ab4"
Nov 03 01:50:49.242193 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:49.242145    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv8zr" podUID="c7a013c2-18a4-4372-adcb-8dfafa673f9c"
Nov 03 01:50:49.378160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:49.377999    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jdk59" event={"ID":"54ba28bb-4bab-4c0d-aa00-8b3bce675f23","Type":"ContainerStarted","Data":"c71b6e03583ced02350063d1da22519015fd8ff07968ce712c61a632fdde2c01"}
Nov 03 01:50:49.378585 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:49.378208    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jdk59" event={"ID":"54ba28bb-4bab-4c0d-aa00-8b3bce675f23","Type":"ContainerStarted","Data":"9403f2c7f8727a5152538bd0f1738f40b4a9dbc0c876c52d34a6f269fa6ae2cf"}
Nov 03 01:50:49.378585 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:49.378260    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jdk59" event={"ID":"54ba28bb-4bab-4c0d-aa00-8b3bce675f23","Type":"ContainerStarted","Data":"99566524a112c39c1810934d6a4ea73cb3933d23b550ad18bffbdb642ed177ad"}
Nov 03 01:50:49.378585 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:49.378311    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jdk59" event={"ID":"54ba28bb-4bab-4c0d-aa00-8b3bce675f23","Type":"ContainerStarted","Data":"a97f939284df84586a6d7c69223aa5226d85d4734dc986c2a604342e27196b30"}
Nov 03 01:50:49.378585 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:49.378353    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jdk59" event={"ID":"54ba28bb-4bab-4c0d-aa00-8b3bce675f23","Type":"ContainerStarted","Data":"4880c4e4742a59710b9eccbb1fe9bce773102ef2f7644b4e1117bc5a5d95873b"}
Nov 03 01:50:49.378585 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:49.378396    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jdk59" event={"ID":"54ba28bb-4bab-4c0d-aa00-8b3bce675f23","Type":"ContainerStarted","Data":"1dd4e49fd8a644c29780517d07718d45a18d5c88c5424f9e1ca3483c4999747e"}
Nov 03 01:50:51.242020 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:51.241946    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:51.242020 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:51.241970    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:51.242390 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:51.242081    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv8zr" podUID="c7a013c2-18a4-4372-adcb-8dfafa673f9c"
Nov 03 01:50:51.242390 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:51.242115    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:50:51.242390 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:51.242162    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mxg52" podUID="79fe02fa-99ce-4222-b569-6b323df47ab4"
Nov 03 01:50:51.242390 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:51.242205    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdr8p" podUID="42a56b4b-283d-4e63-bb6a-59c5d5df0c2c"
Nov 03 01:50:51.388972 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:51.388941    2571 generic.go:356] "Generic (PLEG): container finished" podID="e49068d1-9ae4-47b3-9a07-3d695a0a2402" containerID="51604b80f0fd685dce59fddb01b34582be172c46bc9c4ee2a290ca691b1679d1" exitCode=0
Nov 03 01:50:51.389129 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:51.389030    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7qsp" event={"ID":"e49068d1-9ae4-47b3-9a07-3d695a0a2402","Type":"ContainerDied","Data":"51604b80f0fd685dce59fddb01b34582be172c46bc9c4ee2a290ca691b1679d1"}
Nov 03 01:50:51.392244 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:51.392227    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jdk59" event={"ID":"54ba28bb-4bab-4c0d-aa00-8b3bce675f23","Type":"ContainerStarted","Data":"3fecaba1a05e72c8e9c6be0db041ecec584d3ad7528a423f560a230cab080f15"}
Nov 03 01:50:52.397509 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:52.397479    2571 generic.go:356] "Generic (PLEG): container finished" podID="e49068d1-9ae4-47b3-9a07-3d695a0a2402" containerID="308252bb402cfd85e2daddb0bb79145bb24b19b75eeaf6c813430d3a7fe4fdc6" exitCode=0
Nov 03 01:50:52.398164 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:52.397536    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7qsp" event={"ID":"e49068d1-9ae4-47b3-9a07-3d695a0a2402","Type":"ContainerDied","Data":"308252bb402cfd85e2daddb0bb79145bb24b19b75eeaf6c813430d3a7fe4fdc6"}
Nov 03 01:50:53.241804 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:53.241780    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:53.241916 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:53.241881    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:53.241916 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:53.241905    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:50:53.241916 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:53.241898    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv8zr" podUID="c7a013c2-18a4-4372-adcb-8dfafa673f9c"
Nov 03 01:50:53.242010 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:53.241958    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mxg52" podUID="79fe02fa-99ce-4222-b569-6b323df47ab4"
Nov 03 01:50:53.242010 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:53.242005    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdr8p" podUID="42a56b4b-283d-4e63-bb6a-59c5d5df0c2c"
Nov 03 01:50:53.403626 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:53.403586    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7qsp" event={"ID":"e49068d1-9ae4-47b3-9a07-3d695a0a2402","Type":"ContainerStarted","Data":"bd17a3d8ca8f491dd07eeb2c343328f989d75131a8b76ecaae6af053f76ca236"}
Nov 03 01:50:53.407234 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:53.407205    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jdk59" event={"ID":"54ba28bb-4bab-4c0d-aa00-8b3bce675f23","Type":"ContainerStarted","Data":"a4b6fbad6c9aed736959d1fa2bb60c537d8a68eefdd35842c9586cb21af8aa12"}
Nov 03 01:50:53.407476 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:53.407450    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:53.407541 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:53.407511    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:53.407579 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:53.407543    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:53.422684 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:53.422662    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:53.422795 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:53.422780    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:50:53.611111 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:53.611067    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d7qsp" podStartSLOduration=8.714878689 podStartE2EDuration="23.611054238s" totalImagesPullingTime="14.896175549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:53.570433481 +0000 UTC m=+23.858004289" watchObservedRunningTime="2026-11-03 01:50:53.611054238 +0000 UTC m=+23.898625032"
Nov 03 01:50:53.691513 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:53.691463    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jdk59" podStartSLOduration=7.65021555 podStartE2EDuration="23.691449964s" totalImagesPullingTime="16.041234414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:53.690912043 +0000 UTC m=+23.978482838" watchObservedRunningTime="2026-11-03 01:50:53.691449964 +0000 UTC m=+23.979020775"
Nov 03 01:50:55.241853 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:55.241813    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:55.241853 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:55.241840    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:55.242371 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:55.241937    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv8zr" podUID="c7a013c2-18a4-4372-adcb-8dfafa673f9c"
Nov 03 01:50:55.242371 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:55.242028    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mxg52" podUID="79fe02fa-99ce-4222-b569-6b323df47ab4"
Nov 03 01:50:55.242371 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:55.242051    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:50:55.242371 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:55.242123    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdr8p" podUID="42a56b4b-283d-4e63-bb6a-59c5d5df0c2c"
Nov 03 01:50:55.962336 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:55.962296    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mxg52"]
Nov 03 01:50:55.962510 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:55.962422    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:55.962567 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:55.962513    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mxg52" podUID="79fe02fa-99ce-4222-b569-6b323df47ab4"
Nov 03 01:50:55.965295 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:55.965269    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mdr8p"]
Nov 03 01:50:55.965411 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:55.965361    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:50:55.965479 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:55.965456    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdr8p" podUID="42a56b4b-283d-4e63-bb6a-59c5d5df0c2c"
Nov 03 01:50:56.012846 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:56.012821    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xv8zr"]
Nov 03 01:50:56.012940 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:56.012911    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:56.013031 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:56.013014    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv8zr" podUID="c7a013c2-18a4-4372-adcb-8dfafa673f9c"
Nov 03 01:50:57.241399 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:57.241372    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:57.241808 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:57.241471    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv8zr" podUID="c7a013c2-18a4-4372-adcb-8dfafa673f9c"
Nov 03 01:50:58.242377 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:58.242119    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:50:58.242377 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:58.242176    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:50:58.242916 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:58.242375    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mxg52" podUID="79fe02fa-99ce-4222-b569-6b323df47ab4"
Nov 03 01:50:58.242916 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:58.242437    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdr8p" podUID="42a56b4b-283d-4e63-bb6a-59c5d5df0c2c"
Nov 03 01:50:59.241820 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:50:59.241751    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:50:59.241947 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:50:59.241853    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv8zr" podUID="c7a013c2-18a4-4372-adcb-8dfafa673f9c"
Nov 03 01:51:00.241972 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:00.241899    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:51:00.242385 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:00.241986    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mxg52" podUID="79fe02fa-99ce-4222-b569-6b323df47ab4"
Nov 03 01:51:00.242385 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:00.242063    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:51:00.242975 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:00.242579    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdr8p" podUID="42a56b4b-283d-4e63-bb6a-59c5d5df0c2c"
Nov 03 01:51:01.242116 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:01.242088    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:51:01.242502 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:01.242189    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xv8zr" podUID="c7a013c2-18a4-4372-adcb-8dfafa673f9c"
Nov 03 01:51:02.097932 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.097908    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeReady"
Nov 03 01:51:02.098096 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.098002    2571 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 01:51:02.211581 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.211547    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rcv8q"]
Nov 03 01:51:02.229832 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.229806    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jqkfr"]
Nov 03 01:51:02.229973 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.229946    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-rcv8q"
Nov 03 01:51:02.249779 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.249757    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l87nq"]
Nov 03 01:51:02.250111 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.249852    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jqkfr"
Nov 03 01:51:02.250111 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.249865    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:51:02.250111 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.250058    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:51:02.272165 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.272144    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-l87nq"
Nov 03 01:51:02.276266 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.276246    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn"]
Nov 03 01:51:02.296124 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.296097    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr"]
Nov 03 01:51:02.296257 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.296233    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn"
Nov 03 01:51:02.305902 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.305885    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 01:51:02.309247 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.309220    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xbq7\" (UniqueName: \"kubernetes.io/projected/e11ea02d-ac52-4318-9fdd-95c0bb171aed-kube-api-access-2xbq7\") pod \"kas-connection-checker-685895c7c9-jqkfr\" (UID: \"e11ea02d-ac52-4318-9fdd-95c0bb171aed\") " pod="kube-system/kas-connection-checker-685895c7c9-jqkfr"
Nov 03 01:51:02.309348 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.309261    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btsvv\" (UniqueName: \"kubernetes.io/projected/3a6655a6-9079-4927-8e6c-09d075de438c-kube-api-access-btsvv\") pod \"kas-connection-checker-685895c7c9-rcv8q\" (UID: \"3a6655a6-9079-4927-8e6c-09d075de438c\") " pod="kube-system/kas-connection-checker-685895c7c9-rcv8q"
Nov 03 01:51:02.312582 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.312558    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zxsvp\""
Nov 03 01:51:02.312823 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.312803    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 01:51:02.312823 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.312800    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 01:51:02.312974 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.312802    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wpc48\""
Nov 03 01:51:02.312974 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.312822    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 01:51:02.312974 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.312871    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 01:51:02.317816 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.317798    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6bcfc5667d-w8v7m"]
Nov 03 01:51:02.317945 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.317925    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr"
Nov 03 01:51:02.334540 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.334521    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-55bnn"]
Nov 03 01:51:02.334647 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.334633    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.362803 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.362737    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7875dd7849-ptjkr"]
Nov 03 01:51:02.362946 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.362844    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn"
Nov 03 01:51:02.379844 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.379824    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-gqz7v"]
Nov 03 01:51:02.379964 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.379925    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:02.406130 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.406108    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 01:51:02.406214 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.406147    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 01:51:02.406456 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.406430    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 01:51:02.406598 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.406460    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 01:51:02.406598 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.406469    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 01:51:02.406598 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.406473    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-kgn4w\""
Nov 03 01:51:02.406598 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.406494    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 01:51:02.406598 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.406501    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 01:51:02.406598 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.406510    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 01:51:02.406598 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.406532    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bnxtn\""
Nov 03 01:51:02.406598 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.406535    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 01:51:02.407056 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.406656    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-m9xwl\""
Nov 03 01:51:02.407056 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.406868    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 01:51:02.407056 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.406867    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 01:51:02.407983 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.407969    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"]
Nov 03 01:51:02.408097 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.408083    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:02.410080 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.410054    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/108d9224-ccd3-4925-a65b-0d59e938ff72-ca-trust-extracted\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.410176 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.410100    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg5wd\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-kube-api-access-cg5wd\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.410176 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.410132    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ln9h\" (UniqueName: \"kubernetes.io/projected/38de413a-873a-4288-85f5-c453581a53d6-kube-api-access-9ln9h\") pod \"kas-connection-checker-685895c7c9-l87nq\" (UID: \"38de413a-873a-4288-85f5-c453581a53d6\") " pod="kube-system/kas-connection-checker-685895c7c9-l87nq"
Nov 03 01:51:02.410176 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.410162    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/15face97-689e-4fef-86c0-17a2a85829db-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t59nn\" (UID: \"15face97-689e-4fef-86c0-17a2a85829db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn"
Nov 03 01:51:02.410343 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.410202    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/108d9224-ccd3-4925-a65b-0d59e938ff72-installation-pull-secrets\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.410343 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.410285    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/baf5281f-5938-4417-b22f-2644034da7ef-serving-cert\") pod \"service-ca-operator-544d749c8c-nw7sr\" (UID: \"baf5281f-5938-4417-b22f-2644034da7ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr"
Nov 03 01:51:02.410343 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.410314    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/baf5281f-5938-4417-b22f-2644034da7ef-config\") pod \"service-ca-operator-544d749c8c-nw7sr\" (UID: \"baf5281f-5938-4417-b22f-2644034da7ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr"
Nov 03 01:51:02.410481 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.410360    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xbq7\" (UniqueName: \"kubernetes.io/projected/e11ea02d-ac52-4318-9fdd-95c0bb171aed-kube-api-access-2xbq7\") pod \"kas-connection-checker-685895c7c9-jqkfr\" (UID: \"e11ea02d-ac52-4318-9fdd-95c0bb171aed\") " pod="kube-system/kas-connection-checker-685895c7c9-jqkfr"
Nov 03 01:51:02.410481 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.410395    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/108d9224-ccd3-4925-a65b-0d59e938ff72-image-registry-private-configuration\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.410481 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.410433    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btsvv\" (UniqueName: \"kubernetes.io/projected/3a6655a6-9079-4927-8e6c-09d075de438c-kube-api-access-btsvv\") pod \"kas-connection-checker-685895c7c9-rcv8q\" (UID: \"3a6655a6-9079-4927-8e6c-09d075de438c\") " pod="kube-system/kas-connection-checker-685895c7c9-rcv8q"
Nov 03 01:51:02.410617 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.410530    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc9gg\" (UniqueName: \"kubernetes.io/projected/baf5281f-5938-4417-b22f-2644034da7ef-kube-api-access-mc9gg\") pod \"service-ca-operator-544d749c8c-nw7sr\" (UID: \"baf5281f-5938-4417-b22f-2644034da7ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr"
Nov 03 01:51:02.410848 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.410826    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-certificates\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.410993 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.410977    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/108d9224-ccd3-4925-a65b-0d59e938ff72-trusted-ca\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.411114 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.411092    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqsmb\" (UniqueName: \"kubernetes.io/projected/15face97-689e-4fef-86c0-17a2a85829db-kube-api-access-zqsmb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t59nn\" (UID: \"15face97-689e-4fef-86c0-17a2a85829db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn"
Nov 03 01:51:02.411207 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.411131    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/15face97-689e-4fef-86c0-17a2a85829db-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t59nn\" (UID: \"15face97-689e-4fef-86c0-17a2a85829db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn"
Nov 03 01:51:02.411207 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.411165    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.411207 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.411197    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-bound-sa-token\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.414283 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.414261    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 01:51:02.415455 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.415435    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-hc9w6\""
Nov 03 01:51:02.415549 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.415455    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 01:51:02.415549 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.415524    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 01:51:02.432872 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.432852    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 01:51:02.432970 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.432873    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-54dhc\""
Nov 03 01:51:02.432970 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.432875    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 01:51:02.432970 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.432909    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 01:51:02.432970 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.432922    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 01:51:02.432970 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.432927    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 01:51:02.432970 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.432943    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 01:51:02.434232 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.434214    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"]
Nov 03 01:51:02.434412 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.434400    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:02.458434 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.458418    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-f5dm4"]
Nov 03 01:51:02.458561 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.458546    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:02.461099 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.461078    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 01:51:02.461191 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.461113    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 01:51:02.461191 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.461122    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mlvnc\""
Nov 03 01:51:02.461311 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.461191    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 01:51:02.461311 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.461218    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 01:51:02.462160 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.462142    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 01:51:02.467912 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.467891    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xbq7\" (UniqueName: \"kubernetes.io/projected/e11ea02d-ac52-4318-9fdd-95c0bb171aed-kube-api-access-2xbq7\") pod \"kas-connection-checker-685895c7c9-jqkfr\" (UID: \"e11ea02d-ac52-4318-9fdd-95c0bb171aed\") " pod="kube-system/kas-connection-checker-685895c7c9-jqkfr"
Nov 03 01:51:02.467989 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.467890    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btsvv\" (UniqueName: \"kubernetes.io/projected/3a6655a6-9079-4927-8e6c-09d075de438c-kube-api-access-btsvv\") pod \"kas-connection-checker-685895c7c9-rcv8q\" (UID: \"3a6655a6-9079-4927-8e6c-09d075de438c\") " pod="kube-system/kas-connection-checker-685895c7c9-rcv8q"
Nov 03 01:51:02.477034 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.477015    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55fd5cff67-bb76l"]
Nov 03 01:51:02.477157 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.477142    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-f5dm4"
Nov 03 01:51:02.494833 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.494809    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"]
Nov 03 01:51:02.494934 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.494886    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.508606 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.508588    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 01:51:02.508683 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.508610    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 01:51:02.508979 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.508966    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 01:51:02.509131 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.509117    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 01:51:02.509246 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.509235    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-k78r9\""
Nov 03 01:51:02.511196 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.511182    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 01:51:02.511347 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.511332    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 01:51:02.511433 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.511333    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 01:51:02.511433 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.511373    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 01:51:02.511433 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.511377    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-87fvc\""
Nov 03 01:51:02.511580 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.511473    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/baf5281f-5938-4417-b22f-2644034da7ef-config\") pod \"service-ca-operator-544d749c8c-nw7sr\" (UID: \"baf5281f-5938-4417-b22f-2644034da7ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr"
Nov 03 01:51:02.511580 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.511496    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkw2h\" (UniqueName: \"kubernetes.io/projected/ef90f36a-279e-4bfa-a97d-70132a266670-kube-api-access-kkw2h\") pod \"cluster-samples-operator-6ddb6c8b94-qtpnh\" (UID: \"ef90f36a-279e-4bfa-a97d-70132a266670\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:02.511580 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.511551    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzfk9\" (UniqueName: \"kubernetes.io/projected/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-kube-api-access-rzfk9\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:02.511748 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.511588    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-qtpnh\" (UID: \"ef90f36a-279e-4bfa-a97d-70132a266670\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:02.511748 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.511615    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/354d7018-887a-497f-b74f-195c8c88e184-service-ca-bundle\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:02.511748 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.511633    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/108d9224-ccd3-4925-a65b-0d59e938ff72-image-registry-private-configuration\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.511748 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.511710    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-z28hp"]
Nov 03 01:51:02.511945 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.511756    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qtpnh\" (UID: \"ef90f36a-279e-4bfa-a97d-70132a266670\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:02.511945 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.511837    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc9gg\" (UniqueName: \"kubernetes.io/projected/baf5281f-5938-4417-b22f-2644034da7ef-kube-api-access-mc9gg\") pod \"service-ca-operator-544d749c8c-nw7sr\" (UID: \"baf5281f-5938-4417-b22f-2644034da7ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr"
Nov 03 01:51:02.511945 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.511865    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-default-certificate\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:02.511945 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.511913    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/354d7018-887a-497f-b74f-195c8c88e184-snapshots\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:02.512143 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.511965    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/15face97-689e-4fef-86c0-17a2a85829db-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t59nn\" (UID: \"15face97-689e-4fef-86c0-17a2a85829db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn"
Nov 03 01:51:02.512143 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.511985    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/354d7018-887a-497f-b74f-195c8c88e184-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:02.512143 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.512026    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg5wd\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-kube-api-access-cg5wd\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.512143 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.512080    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:02.512143 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.512114    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/354d7018-887a-497f-b74f-195c8c88e184-tmp\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:02.512378 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.512147    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-55bnn\" (UID: \"0110b28b-ec04-4f65-bc3d-2e82f88b30d7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn"
Nov 03 01:51:02.512378 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.512177    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-certificates\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.512378 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.512208    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-stats-auth\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:02.512378 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.512234    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/108d9224-ccd3-4925-a65b-0d59e938ff72-trusted-ca\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.512378 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.512259    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqsmb\" (UniqueName: \"kubernetes.io/projected/15face97-689e-4fef-86c0-17a2a85829db-kube-api-access-zqsmb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t59nn\" (UID: \"15face97-689e-4fef-86c0-17a2a85829db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn"
Nov 03 01:51:02.512378 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.512289    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.512378 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.512313    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-bound-sa-token\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.512378 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.512338    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg6dt\" (UniqueName: \"kubernetes.io/projected/354d7018-887a-497f-b74f-195c8c88e184-kube-api-access-fg6dt\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:02.512378 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.512367    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/354d7018-887a-497f-b74f-195c8c88e184-serving-cert\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:02.512839 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.512395    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/108d9224-ccd3-4925-a65b-0d59e938ff72-ca-trust-extracted\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.512839 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.512422    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ln9h\" (UniqueName: \"kubernetes.io/projected/38de413a-873a-4288-85f5-c453581a53d6-kube-api-access-9ln9h\") pod \"kas-connection-checker-685895c7c9-l87nq\" (UID: \"38de413a-873a-4288-85f5-c453581a53d6\") " pod="kube-system/kas-connection-checker-685895c7c9-l87nq"
Nov 03 01:51:02.512839 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.512451    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/15face97-689e-4fef-86c0-17a2a85829db-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t59nn\" (UID: \"15face97-689e-4fef-86c0-17a2a85829db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn"
Nov 03 01:51:02.512839 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.512480    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:02.512839 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.512509    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/108d9224-ccd3-4925-a65b-0d59e938ff72-installation-pull-secrets\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.512839 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.512536    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-55bnn\" (UID: \"0110b28b-ec04-4f65-bc3d-2e82f88b30d7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn"
Nov 03 01:51:02.512839 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.512565    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/baf5281f-5938-4417-b22f-2644034da7ef-serving-cert\") pod \"service-ca-operator-544d749c8c-nw7sr\" (UID: \"baf5281f-5938-4417-b22f-2644034da7ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr"
Nov 03 01:51:02.512839 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.512591    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:02.512839 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.512628    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/baf5281f-5938-4417-b22f-2644034da7ef-config\") pod \"service-ca-operator-544d749c8c-nw7sr\" (UID: \"baf5281f-5938-4417-b22f-2644034da7ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr"
Nov 03 01:51:02.512839 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:02.512784    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 01:51:02.512839 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:02.512801    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bcfc5667d-w8v7m: secret "image-registry-tls" not found
Nov 03 01:51:02.513347 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:02.512849    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls podName:108d9224-ccd3-4925-a65b-0d59e938ff72 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:03.012831283 +0000 UTC m=+33.300402093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls") pod "image-registry-6bcfc5667d-w8v7m" (UID: "108d9224-ccd3-4925-a65b-0d59e938ff72") : secret "image-registry-tls" not found
Nov 03 01:51:02.513347 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.512864    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 01:51:02.513456 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.513410    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/108d9224-ccd3-4925-a65b-0d59e938ff72-ca-trust-extracted\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.513975 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.513952    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/15face97-689e-4fef-86c0-17a2a85829db-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t59nn\" (UID: \"15face97-689e-4fef-86c0-17a2a85829db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn"
Nov 03 01:51:02.514087 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.514059    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-certificates\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.514376 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.514355    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/108d9224-ccd3-4925-a65b-0d59e938ff72-trusted-ca\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.514497 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.514471    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/108d9224-ccd3-4925-a65b-0d59e938ff72-image-registry-private-configuration\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.515782 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.515762    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/baf5281f-5938-4417-b22f-2644034da7ef-serving-cert\") pod \"service-ca-operator-544d749c8c-nw7sr\" (UID: \"baf5281f-5938-4417-b22f-2644034da7ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr"
Nov 03 01:51:02.515857 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.515804    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/15face97-689e-4fef-86c0-17a2a85829db-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t59nn\" (UID: \"15face97-689e-4fef-86c0-17a2a85829db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn"
Nov 03 01:51:02.515922 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.515907    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/108d9224-ccd3-4925-a65b-0d59e938ff72-installation-pull-secrets\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.529776 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.529757    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jfs9g\""
Nov 03 01:51:02.529853 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.529836    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 01:51:02.529887 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.529860    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 01:51:02.543951 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.543929    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7t99k"]
Nov 03 01:51:02.544046 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.544028    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z28hp"
Nov 03 01:51:02.556763 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.556747    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-rcv8q"
Nov 03 01:51:02.560100 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.560085    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w4h82"]
Nov 03 01:51:02.560201 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.560187    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7t99k"
Nov 03 01:51:02.564690 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.564675    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 01:51:02.564804 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.564790    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9tjtm\""
Nov 03 01:51:02.565100 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.565079    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 01:51:02.565882 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.565864    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 01:51:02.566037 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.566025    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 01:51:02.569035 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.568922    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc9gg\" (UniqueName: \"kubernetes.io/projected/baf5281f-5938-4417-b22f-2644034da7ef-kube-api-access-mc9gg\") pod \"service-ca-operator-544d749c8c-nw7sr\" (UID: \"baf5281f-5938-4417-b22f-2644034da7ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr"
Nov 03 01:51:02.570373 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.570349    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ln9h\" (UniqueName: \"kubernetes.io/projected/38de413a-873a-4288-85f5-c453581a53d6-kube-api-access-9ln9h\") pod \"kas-connection-checker-685895c7c9-l87nq\" (UID: \"38de413a-873a-4288-85f5-c453581a53d6\") " pod="kube-system/kas-connection-checker-685895c7c9-l87nq"
Nov 03 01:51:02.571099 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.571069    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg5wd\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-kube-api-access-cg5wd\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.573713 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.572495    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqsmb\" (UniqueName: \"kubernetes.io/projected/15face97-689e-4fef-86c0-17a2a85829db-kube-api-access-zqsmb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t59nn\" (UID: \"15face97-689e-4fef-86c0-17a2a85829db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn"
Nov 03 01:51:02.575180 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.575161    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-bound-sa-token\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:02.576115 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.576098    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jqkfr"
Nov 03 01:51:02.581830 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.581811    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rcv8q"]
Nov 03 01:51:02.581912 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.581848    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jqkfr"]
Nov 03 01:51:02.581912 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.581871    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l87nq"]
Nov 03 01:51:02.581912 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.581892    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr"]
Nov 03 01:51:02.581912 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.581911    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn"]
Nov 03 01:51:02.582106 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.581930    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bcfc5667d-w8v7m"]
Nov 03 01:51:02.582106 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.581937    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:02.582106 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.581950    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-f5dm4"]
Nov 03 01:51:02.582106 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.581969    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-gqz7v"]
Nov 03 01:51:02.582106 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.581987    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7t99k"]
Nov 03 01:51:02.582106 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.582009    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"]
Nov 03 01:51:02.582106 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.582035    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-z28hp"]
Nov 03 01:51:02.582106 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.582055    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w4h82"]
Nov 03 01:51:02.582106 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.582078    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55fd5cff67-bb76l"]
Nov 03 01:51:02.582106 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.582097    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-55bnn"]
Nov 03 01:51:02.582461 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.582118    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"]
Nov 03 01:51:02.582461 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.582138    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"]
Nov 03 01:51:02.582461 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.582157    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7875dd7849-ptjkr"]
Nov 03 01:51:02.609537 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.609515    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rwql9\""
Nov 03 01:51:02.612561 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.612545    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 01:51:02.612956 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.612910    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 01:51:02.613005 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.612957    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 01:51:02.613005 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.612974    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-v5ks7\""
Nov 03 01:51:02.613070 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613037    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.613108 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613073    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c546cea-b474-44cf-84a5-b89784bda1d6-installation-pull-secrets\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.613108 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613101    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t8q9\" (UniqueName: \"kubernetes.io/projected/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-kube-api-access-6t8q9\") pod \"console-operator-7cd56b8dbc-rjmbd\" (UID: \"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:02.613186 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613129    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvkdd\" (UniqueName: \"kubernetes.io/projected/665f9d02-067e-4506-9793-cc7f17eb95bb-kube-api-access-cvkdd\") pod \"network-check-source-6f78d59f99-z28hp\" (UID: \"665f9d02-067e-4506-9793-cc7f17eb95bb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z28hp"
Nov 03 01:51:02.613220 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613206    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/354d7018-887a-497f-b74f-195c8c88e184-tmp\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:02.613286 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613270    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f4425c17-5cc6-4afc-922c-d74694805196-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:02.613317 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613306    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-55bnn\" (UID: \"0110b28b-ec04-4f65-bc3d-2e82f88b30d7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn"
Nov 03 01:51:02.613366 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613352    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-bound-sa-token\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.613465 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613449    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/354d7018-887a-497f-b74f-195c8c88e184-tmp\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:02.613538 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613522    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-stats-auth\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:02.613579 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613568    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85997\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-kube-api-access-85997\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.613625 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613611    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg6dt\" (UniqueName: \"kubernetes.io/projected/354d7018-887a-497f-b74f-195c8c88e184-kube-api-access-fg6dt\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:02.613659 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613640    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/354d7018-887a-497f-b74f-195c8c88e184-serving-cert\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:02.613709 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613671    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c546cea-b474-44cf-84a5-b89784bda1d6-trusted-ca\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.613746 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613720    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-trusted-ca\") pod \"console-operator-7cd56b8dbc-rjmbd\" (UID: \"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:02.613777 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613749    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:02.613816 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613776    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c546cea-b474-44cf-84a5-b89784bda1d6-image-registry-private-configuration\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.613816 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613808    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-55bnn\" (UID: \"0110b28b-ec04-4f65-bc3d-2e82f88b30d7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn"
Nov 03 01:51:02.613880 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613837    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:02.613913 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613892    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c546cea-b474-44cf-84a5-b89784bda1d6-ca-trust-extracted\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.613963 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613919    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:02.613963 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613953    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkw2h\" (UniqueName: \"kubernetes.io/projected/ef90f36a-279e-4bfa-a97d-70132a266670-kube-api-access-kkw2h\") pod \"cluster-samples-operator-6ddb6c8b94-qtpnh\" (UID: \"ef90f36a-279e-4bfa-a97d-70132a266670\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:02.614065 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.613986    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzfk9\" (UniqueName: \"kubernetes.io/projected/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-kube-api-access-rzfk9\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:02.614065 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.614013    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-qtpnh\" (UID: \"ef90f36a-279e-4bfa-a97d-70132a266670\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:02.614065 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.614039    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/354d7018-887a-497f-b74f-195c8c88e184-service-ca-bundle\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:02.614212 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.614063    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-config\") pod \"console-operator-7cd56b8dbc-rjmbd\" (UID: \"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:02.614212 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.614090    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qtpnh\" (UID: \"ef90f36a-279e-4bfa-a97d-70132a266670\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:02.614212 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.614127    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-default-certificate\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:02.614212 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.614149    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/354d7018-887a-497f-b74f-195c8c88e184-snapshots\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:02.614212 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:02.614167    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 01:51:02.614212 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.614168    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-certificates\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.614212 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:02.614215    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls podName:ef90f36a-279e-4bfa-a97d-70132a266670 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:03.114200753 +0000 UTC m=+33.401771557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qtpnh" (UID: "ef90f36a-279e-4bfa-a97d-70132a266670") : secret "samples-operator-tls" not found
Nov 03 01:51:02.614554 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.614404    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxvzg\" (UniqueName: \"kubernetes.io/projected/48604fe3-9728-461e-ae37-8208b424e149-kube-api-access-zxvzg\") pod \"volume-data-source-validator-58c9f5f46d-f5dm4\" (UID: \"48604fe3-9728-461e-ae37-8208b424e149\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-f5dm4"
Nov 03 01:51:02.614554 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.614431    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/354d7018-887a-497f-b74f-195c8c88e184-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:02.614554 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.614463    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n62b\" (UniqueName: \"kubernetes.io/projected/f4425c17-5cc6-4afc-922c-d74694805196-kube-api-access-8n62b\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:02.614554 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.614483    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-serving-cert\") pod \"console-operator-7cd56b8dbc-rjmbd\" (UID: \"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:02.614964 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:02.614948    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 01:51:02.615037 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:02.615009    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs podName:7a9e57fd-a6e6-4ddb-aa08-ac1390403544 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:03.114993076 +0000 UTC m=+33.402563872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs") pod "router-default-7875dd7849-ptjkr" (UID: "7a9e57fd-a6e6-4ddb-aa08-ac1390403544") : secret "router-metrics-certs-default" not found
Nov 03 01:51:02.615124 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:02.615111    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle podName:7a9e57fd-a6e6-4ddb-aa08-ac1390403544 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:03.115102524 +0000 UTC m=+33.402673296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle") pod "router-default-7875dd7849-ptjkr" (UID: "7a9e57fd-a6e6-4ddb-aa08-ac1390403544") : configmap references non-existent config key: service-ca.crt
Nov 03 01:51:02.615213 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:02.615201    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 01:51:02.615327 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:02.615250    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert podName:0110b28b-ec04-4f65-bc3d-2e82f88b30d7 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:03.115238937 +0000 UTC m=+33.402809747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-55bnn" (UID: "0110b28b-ec04-4f65-bc3d-2e82f88b30d7") : secret "networking-console-plugin-cert" not found
Nov 03 01:51:02.615327 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.615248    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-qtpnh\" (UID: \"ef90f36a-279e-4bfa-a97d-70132a266670\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:02.615327 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.615275    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-55bnn\" (UID: \"0110b28b-ec04-4f65-bc3d-2e82f88b30d7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn"
Nov 03 01:51:02.615625 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.615606    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/354d7018-887a-497f-b74f-195c8c88e184-service-ca-bundle\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:02.615743 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.615727    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/354d7018-887a-497f-b74f-195c8c88e184-snapshots\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:02.616109 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.616087    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/354d7018-887a-497f-b74f-195c8c88e184-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:02.617158 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.617138    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-default-certificate\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:02.617300 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.617249    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-stats-auth\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:02.619418 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.618455    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/354d7018-887a-497f-b74f-195c8c88e184-serving-cert\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:02.634789 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.634747    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 01:51:02.634912 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.634842    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bsx6s\""
Nov 03 01:51:02.634912 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.634884    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 01:51:02.641013 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.640970    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn"
Nov 03 01:51:02.663619 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.663600    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-l87nq"
Nov 03 01:51:02.676271 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.676206    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg6dt\" (UniqueName: \"kubernetes.io/projected/354d7018-887a-497f-b74f-195c8c88e184-kube-api-access-fg6dt\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:02.677721 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.677314    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkw2h\" (UniqueName: \"kubernetes.io/projected/ef90f36a-279e-4bfa-a97d-70132a266670-kube-api-access-kkw2h\") pod \"cluster-samples-operator-6ddb6c8b94-qtpnh\" (UID: \"ef90f36a-279e-4bfa-a97d-70132a266670\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:02.678841 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.677959    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzfk9\" (UniqueName: \"kubernetes.io/projected/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-kube-api-access-rzfk9\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:02.698577 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.698550    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr"
Nov 03 01:51:02.716039 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716009    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n62b\" (UniqueName: \"kubernetes.io/projected/f4425c17-5cc6-4afc-922c-d74694805196-kube-api-access-8n62b\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:02.716146 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716058    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-serving-cert\") pod \"console-operator-7cd56b8dbc-rjmbd\" (UID: \"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:02.716146 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716090    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.716146 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716115    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c546cea-b474-44cf-84a5-b89784bda1d6-installation-pull-secrets\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.716297 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716143    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t8q9\" (UniqueName: \"kubernetes.io/projected/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-kube-api-access-6t8q9\") pod \"console-operator-7cd56b8dbc-rjmbd\" (UID: \"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:02.716297 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716170    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvkdd\" (UniqueName: \"kubernetes.io/projected/665f9d02-067e-4506-9793-cc7f17eb95bb-kube-api-access-cvkdd\") pod \"network-check-source-6f78d59f99-z28hp\" (UID: \"665f9d02-067e-4506-9793-cc7f17eb95bb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z28hp"
Nov 03 01:51:02.716297 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716227    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f4425c17-5cc6-4afc-922c-d74694805196-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:02.716297 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716256    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2555270f-a2ea-4d11-8dff-d130f9786e3d-config-volume\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:02.716297 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716291    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-bound-sa-token\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.716532 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716323    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b25bd7be-290a-499a-a81d-c005793eace2-cert\") pod \"ingress-canary-7t99k\" (UID: \"b25bd7be-290a-499a-a81d-c005793eace2\") " pod="openshift-ingress-canary/ingress-canary-7t99k"
Nov 03 01:51:02.716532 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716357    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85997\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-kube-api-access-85997\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.716532 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716417    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c546cea-b474-44cf-84a5-b89784bda1d6-trusted-ca\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.716532 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716442    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-trusted-ca\") pod \"console-operator-7cd56b8dbc-rjmbd\" (UID: \"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:02.716532 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716485    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c546cea-b474-44cf-84a5-b89784bda1d6-image-registry-private-configuration\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.716786 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716554    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c546cea-b474-44cf-84a5-b89784bda1d6-ca-trust-extracted\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.716786 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716582    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:02.716786 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716615    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5lg2\" (UniqueName: \"kubernetes.io/projected/2555270f-a2ea-4d11-8dff-d130f9786e3d-kube-api-access-q5lg2\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:02.716786 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716655    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-config\") pod \"console-operator-7cd56b8dbc-rjmbd\" (UID: \"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:02.716786 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716679    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c6wq\" (UniqueName: \"kubernetes.io/projected/b25bd7be-290a-499a-a81d-c005793eace2-kube-api-access-8c6wq\") pod \"ingress-canary-7t99k\" (UID: \"b25bd7be-290a-499a-a81d-c005793eace2\") " pod="openshift-ingress-canary/ingress-canary-7t99k"
Nov 03 01:51:02.716786 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716742    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-certificates\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.716786 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716770    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxvzg\" (UniqueName: \"kubernetes.io/projected/48604fe3-9728-461e-ae37-8208b424e149-kube-api-access-zxvzg\") pod \"volume-data-source-validator-58c9f5f46d-f5dm4\" (UID: \"48604fe3-9728-461e-ae37-8208b424e149\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-f5dm4"
Nov 03 01:51:02.717116 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716800    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2555270f-a2ea-4d11-8dff-d130f9786e3d-tmp-dir\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:02.717116 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.716838    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:02.718554 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.718260    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-trusted-ca\") pod \"console-operator-7cd56b8dbc-rjmbd\" (UID: \"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:02.719541 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.719519    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c546cea-b474-44cf-84a5-b89784bda1d6-trusted-ca\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.719626 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:02.719576    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 01:51:02.719676 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:02.719645    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls podName:f4425c17-5cc6-4afc-922c-d74694805196 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:03.219626986 +0000 UTC m=+33.507197798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-g8wz8" (UID: "f4425c17-5cc6-4afc-922c-d74694805196") : secret "cluster-monitoring-operator-tls" not found
Nov 03 01:51:02.720519 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.720076    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-config\") pod \"console-operator-7cd56b8dbc-rjmbd\" (UID: \"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:02.720519 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.720234    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f4425c17-5cc6-4afc-922c-d74694805196-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:02.720519 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:02.720353    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 01:51:02.720519 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:02.720369    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55fd5cff67-bb76l: secret "image-registry-tls" not found
Nov 03 01:51:02.720519 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:02.720415    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls podName:9c546cea-b474-44cf-84a5-b89784bda1d6 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:03.220399207 +0000 UTC m=+33.507969985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls") pod "image-registry-55fd5cff67-bb76l" (UID: "9c546cea-b474-44cf-84a5-b89784bda1d6") : secret "image-registry-tls" not found
Nov 03 01:51:02.721669 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.721474    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c546cea-b474-44cf-84a5-b89784bda1d6-ca-trust-extracted\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.721776 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.721709    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-serving-cert\") pod \"console-operator-7cd56b8dbc-rjmbd\" (UID: \"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:02.723995 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.722402    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-certificates\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.725867 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.725614    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c546cea-b474-44cf-84a5-b89784bda1d6-installation-pull-secrets\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.727015 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.726998    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c546cea-b474-44cf-84a5-b89784bda1d6-image-registry-private-configuration\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.760127 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.759097    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:02.775301 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.775273    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n62b\" (UniqueName: \"kubernetes.io/projected/f4425c17-5cc6-4afc-922c-d74694805196-kube-api-access-8n62b\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:02.776185 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.776155    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvkdd\" (UniqueName: \"kubernetes.io/projected/665f9d02-067e-4506-9793-cc7f17eb95bb-kube-api-access-cvkdd\") pod \"network-check-source-6f78d59f99-z28hp\" (UID: \"665f9d02-067e-4506-9793-cc7f17eb95bb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z28hp"
Nov 03 01:51:02.777104 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.777083    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85997\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-kube-api-access-85997\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.790332 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.790303    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxvzg\" (UniqueName: \"kubernetes.io/projected/48604fe3-9728-461e-ae37-8208b424e149-kube-api-access-zxvzg\") pod \"volume-data-source-validator-58c9f5f46d-f5dm4\" (UID: \"48604fe3-9728-461e-ae37-8208b424e149\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-f5dm4"
Nov 03 01:51:02.790785 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.790762    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t8q9\" (UniqueName: \"kubernetes.io/projected/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-kube-api-access-6t8q9\") pod \"console-operator-7cd56b8dbc-rjmbd\" (UID: \"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:02.790841 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.790818    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-bound-sa-token\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:02.812426 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.812400    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-f5dm4"
Nov 03 01:51:02.818435 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.818411    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5lg2\" (UniqueName: \"kubernetes.io/projected/2555270f-a2ea-4d11-8dff-d130f9786e3d-kube-api-access-q5lg2\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:02.818528 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.818448    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c6wq\" (UniqueName: \"kubernetes.io/projected/b25bd7be-290a-499a-a81d-c005793eace2-kube-api-access-8c6wq\") pod \"ingress-canary-7t99k\" (UID: \"b25bd7be-290a-499a-a81d-c005793eace2\") " pod="openshift-ingress-canary/ingress-canary-7t99k"
Nov 03 01:51:02.818528 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.818480    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2555270f-a2ea-4d11-8dff-d130f9786e3d-tmp-dir\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:02.818528 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.818499    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:02.818726 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.818544    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2555270f-a2ea-4d11-8dff-d130f9786e3d-config-volume\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:02.818726 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.818566    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b25bd7be-290a-499a-a81d-c005793eace2-cert\") pod \"ingress-canary-7t99k\" (UID: \"b25bd7be-290a-499a-a81d-c005793eace2\") " pod="openshift-ingress-canary/ingress-canary-7t99k"
Nov 03 01:51:02.819818 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:02.819117    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 01:51:02.819818 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:02.819179    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls podName:2555270f-a2ea-4d11-8dff-d130f9786e3d nodeName:}" failed. No retries permitted until 2026-11-03 01:51:03.31915911 +0000 UTC m=+33.606729899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls") pod "dns-default-w4h82" (UID: "2555270f-a2ea-4d11-8dff-d130f9786e3d") : secret "dns-default-metrics-tls" not found
Nov 03 01:51:02.819818 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.819380    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2555270f-a2ea-4d11-8dff-d130f9786e3d-tmp-dir\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:02.819818 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.819671    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2555270f-a2ea-4d11-8dff-d130f9786e3d-config-volume\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:02.822197 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.821183    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b25bd7be-290a-499a-a81d-c005793eace2-cert\") pod \"ingress-canary-7t99k\" (UID: \"b25bd7be-290a-499a-a81d-c005793eace2\") " pod="openshift-ingress-canary/ingress-canary-7t99k"
Nov 03 01:51:02.852768 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.852554    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:02.878190 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.878110    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c6wq\" (UniqueName: \"kubernetes.io/projected/b25bd7be-290a-499a-a81d-c005793eace2-kube-api-access-8c6wq\") pod \"ingress-canary-7t99k\" (UID: \"b25bd7be-290a-499a-a81d-c005793eace2\") " pod="openshift-ingress-canary/ingress-canary-7t99k"
Nov 03 01:51:02.879803 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.879780    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5lg2\" (UniqueName: \"kubernetes.io/projected/2555270f-a2ea-4d11-8dff-d130f9786e3d-kube-api-access-q5lg2\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:02.890676 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.890519    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7t99k"
Nov 03 01:51:02.919747 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.919359    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z28hp"
Nov 03 01:51:02.995502 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.995430    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rcv8q"]
Nov 03 01:51:02.999066 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:02.998995    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jqkfr"]
Nov 03 01:51:03.003213 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:51:03.003188    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6655a6_9079_4927_8e6c_09d075de438c.slice/crio-ebe7cc246a6948855e1b3fa82fe1be3cdae3ef69f6aa1d6ac06f80a116c5984f WatchSource:0}: Error finding container ebe7cc246a6948855e1b3fa82fe1be3cdae3ef69f6aa1d6ac06f80a116c5984f: Status 404 returned error can't find the container with id ebe7cc246a6948855e1b3fa82fe1be3cdae3ef69f6aa1d6ac06f80a116c5984f
Nov 03 01:51:03.005141 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:51:03.005082    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11ea02d_ac52_4318_9fdd_95c0bb171aed.slice/crio-0c9e32359e3ef256b0e8b244ad8f92a56f410da75819b00fd547aa422d729cb4 WatchSource:0}: Error finding container 0c9e32359e3ef256b0e8b244ad8f92a56f410da75819b00fd547aa422d729cb4: Status 404 returned error can't find the container with id 0c9e32359e3ef256b0e8b244ad8f92a56f410da75819b00fd547aa422d729cb4
Nov 03 01:51:03.026722 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.022168    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:03.026722 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:03.022474    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 01:51:03.026722 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:03.022489    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bcfc5667d-w8v7m: secret "image-registry-tls" not found
Nov 03 01:51:03.026722 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:03.022537    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls podName:108d9224-ccd3-4925-a65b-0d59e938ff72 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:04.022519171 +0000 UTC m=+34.310089962 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls") pod "image-registry-6bcfc5667d-w8v7m" (UID: "108d9224-ccd3-4925-a65b-0d59e938ff72") : secret "image-registry-tls" not found
Nov 03 01:51:03.032571 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.032503    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn"]
Nov 03 01:51:03.037178 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:51:03.037147    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15face97_689e_4fef_86c0_17a2a85829db.slice/crio-9d6e9127f4b7588b09b87bdcb7399d252e4fa32a62cfd57756e7a7246766a2d3 WatchSource:0}: Error finding container 9d6e9127f4b7588b09b87bdcb7399d252e4fa32a62cfd57756e7a7246766a2d3: Status 404 returned error can't find the container with id 9d6e9127f4b7588b09b87bdcb7399d252e4fa32a62cfd57756e7a7246766a2d3
Nov 03 01:51:03.059335 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.059276    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l87nq"]
Nov 03 01:51:03.064389 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:51:03.064352    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38de413a_873a_4288_85f5_c453581a53d6.slice/crio-ee5c4e58b1d36c1d73804b6b742b1354d358c2693d05893eae4ee53d65b508c3 WatchSource:0}: Error finding container ee5c4e58b1d36c1d73804b6b742b1354d358c2693d05893eae4ee53d65b508c3: Status 404 returned error can't find the container with id ee5c4e58b1d36c1d73804b6b742b1354d358c2693d05893eae4ee53d65b508c3
Nov 03 01:51:03.094214 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.094190    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr"]
Nov 03 01:51:03.123651 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.123593    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs\") pod \"network-metrics-daemon-xv8zr\" (UID: \"c7a013c2-18a4-4372-adcb-8dfafa673f9c\") " pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:51:03.123753 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.123671    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:03.123753 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.123721    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-55bnn\" (UID: \"0110b28b-ec04-4f65-bc3d-2e82f88b30d7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn"
Nov 03 01:51:03.123753 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.123748    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:03.123894 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.123765    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:51:03.123894 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.123810    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qtpnh\" (UID: \"ef90f36a-279e-4bfa-a97d-70132a266670\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:03.123963 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:03.123929    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 01:51:03.124001 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:03.123972    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls podName:ef90f36a-279e-4bfa-a97d-70132a266670 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:04.123957371 +0000 UTC m=+34.411528145 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qtpnh" (UID: "ef90f36a-279e-4bfa-a97d-70132a266670") : secret "samples-operator-tls" not found
Nov 03 01:51:03.124058 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:03.124045    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:51:03.124091 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:03.124085    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs podName:c7a013c2-18a4-4372-adcb-8dfafa673f9c nodeName:}" failed. No retries permitted until 2026-11-03 01:51:35.12407393 +0000 UTC m=+65.411644719 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs") pod "network-metrics-daemon-xv8zr" (UID: "c7a013c2-18a4-4372-adcb-8dfafa673f9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:51:03.124164 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:03.124154    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle podName:7a9e57fd-a6e6-4ddb-aa08-ac1390403544 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:04.124143758 +0000 UTC m=+34.411714545 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle") pod "router-default-7875dd7849-ptjkr" (UID: "7a9e57fd-a6e6-4ddb-aa08-ac1390403544") : configmap references non-existent config key: service-ca.crt
Nov 03 01:51:03.124230 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:03.124217    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 01:51:03.124276 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:03.124245    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert podName:0110b28b-ec04-4f65-bc3d-2e82f88b30d7 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:04.12423739 +0000 UTC m=+34.411808164 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-55bnn" (UID: "0110b28b-ec04-4f65-bc3d-2e82f88b30d7") : secret "networking-console-plugin-cert" not found
Nov 03 01:51:03.124311 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:03.124294    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 01:51:03.124351 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:03.124324    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs podName:7a9e57fd-a6e6-4ddb-aa08-ac1390403544 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:04.124314193 +0000 UTC m=+34.411884981 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs") pod "router-default-7875dd7849-ptjkr" (UID: "7a9e57fd-a6e6-4ddb-aa08-ac1390403544") : secret "router-metrics-certs-default" not found
Nov 03 01:51:03.126416 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.126395    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:51:03.134194 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.134173    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-gqz7v"]
Nov 03 01:51:03.184171 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.184146    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-f5dm4"]
Nov 03 01:51:03.188595 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:51:03.188573    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48604fe3_9728_461e_ae37_8208b424e149.slice/crio-16c546e51f74fc98b52aef4f1f105191f19b7af9ee68374cd6c570e88ee5b764 WatchSource:0}: Error finding container 16c546e51f74fc98b52aef4f1f105191f19b7af9ee68374cd6c570e88ee5b764: Status 404 returned error can't find the container with id 16c546e51f74fc98b52aef4f1f105191f19b7af9ee68374cd6c570e88ee5b764
Nov 03 01:51:03.193138 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.193121    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:51:03.225050 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.225029    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:03.225154 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.225062    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq5lq\" (UniqueName: \"kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq\") pod \"network-check-target-mdr8p\" (UID: \"42a56b4b-283d-4e63-bb6a-59c5d5df0c2c\") " pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:51:03.225223 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:03.225197    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 01:51:03.225223 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:03.225216    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55fd5cff67-bb76l: secret "image-registry-tls" not found
Nov 03 01:51:03.225324 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:03.225274    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls podName:9c546cea-b474-44cf-84a5-b89784bda1d6 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:04.225256031 +0000 UTC m=+34.512826817 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls") pod "image-registry-55fd5cff67-bb76l" (UID: "9c546cea-b474-44cf-84a5-b89784bda1d6") : secret "image-registry-tls" not found
Nov 03 01:51:03.225383 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.225319    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:03.225459 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:03.225442    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 01:51:03.225508 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:03.225494    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls podName:f4425c17-5cc6-4afc-922c-d74694805196 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:04.225479619 +0000 UTC m=+34.513050392 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-g8wz8" (UID: "f4425c17-5cc6-4afc-922c-d74694805196") : secret "cluster-monitoring-operator-tls" not found
Nov 03 01:51:03.227341 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.227322    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq5lq\" (UniqueName: \"kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq\") pod \"network-check-target-mdr8p\" (UID: \"42a56b4b-283d-4e63-bb6a-59c5d5df0c2c\") " pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:51:03.237734 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.237711    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"]
Nov 03 01:51:03.241193 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.241152    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:51:03.241298 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:51:03.241210    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8186db5_7ed8_4fd8_95e8_ecd2fd7087fd.slice/crio-68c010f0f76f47a515f05930ada535e464568c6ac669fdd14e110b17f7a77c3a WatchSource:0}: Error finding container 68c010f0f76f47a515f05930ada535e464568c6ac669fdd14e110b17f7a77c3a: Status 404 returned error can't find the container with id 68c010f0f76f47a515f05930ada535e464568c6ac669fdd14e110b17f7a77c3a
Nov 03 01:51:03.293871 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.293839    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 01:51:03.294166 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.293977    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-btxgv\""
Nov 03 01:51:03.300891 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.300867    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-z28hp"]
Nov 03 01:51:03.304393 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:51:03.304375    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665f9d02_067e_4506_9793_cc7f17eb95bb.slice/crio-1d56f450ef97694ee4750e860271e17224a4d8210b40a24273628c3907ad8860 WatchSource:0}: Error finding container 1d56f450ef97694ee4750e860271e17224a4d8210b40a24273628c3907ad8860: Status 404 returned error can't find the container with id 1d56f450ef97694ee4750e860271e17224a4d8210b40a24273628c3907ad8860
Nov 03 01:51:03.326327 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.326305    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:03.326470 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:03.326454    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 01:51:03.326513 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:03.326504    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls podName:2555270f-a2ea-4d11-8dff-d130f9786e3d nodeName:}" failed. No retries permitted until 2026-11-03 01:51:04.326491789 +0000 UTC m=+34.614062563 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls") pod "dns-default-w4h82" (UID: "2555270f-a2ea-4d11-8dff-d130f9786e3d") : secret "dns-default-metrics-tls" not found
Nov 03 01:51:03.407018 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.406899    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7t99k"]
Nov 03 01:51:03.409928 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:51:03.409905    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25bd7be_290a_499a_a81d_c005793eace2.slice/crio-5ecfcc08a256f4626dbbeccb2c8ed8bb53b8fde82e97487eb2f786119c8230c2 WatchSource:0}: Error finding container 5ecfcc08a256f4626dbbeccb2c8ed8bb53b8fde82e97487eb2f786119c8230c2: Status 404 returned error can't find the container with id 5ecfcc08a256f4626dbbeccb2c8ed8bb53b8fde82e97487eb2f786119c8230c2
Nov 03 01:51:03.431349 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.431317    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-rcv8q" event={"ID":"3a6655a6-9079-4927-8e6c-09d075de438c","Type":"ContainerStarted","Data":"c1c94971a1dccbadd65dfad68c3a895b901dbb314c3c896d33715887089615c6"}
Nov 03 01:51:03.431494 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.431369    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-rcv8q" event={"ID":"3a6655a6-9079-4927-8e6c-09d075de438c","Type":"ContainerStarted","Data":"ebe7cc246a6948855e1b3fa82fe1be3cdae3ef69f6aa1d6ac06f80a116c5984f"}
Nov 03 01:51:03.432740 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.432716    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr" event={"ID":"baf5281f-5938-4417-b22f-2644034da7ef","Type":"ContainerStarted","Data":"c1d497aa54f9d2bfadefb60b3ecd9f69712b6dcc0d918a57a7de9b2170079aa3"}
Nov 03 01:51:03.433773 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.433750    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn" event={"ID":"15face97-689e-4fef-86c0-17a2a85829db","Type":"ContainerStarted","Data":"9d6e9127f4b7588b09b87bdcb7399d252e4fa32a62cfd57756e7a7246766a2d3"}
Nov 03 01:51:03.435035 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.435014    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jqkfr" event={"ID":"e11ea02d-ac52-4318-9fdd-95c0bb171aed","Type":"ContainerStarted","Data":"510062eafc2e7ddc28095d9e9927c97277628278eff2b995ebe7b52858dd8733"}
Nov 03 01:51:03.435117 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.435050    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jqkfr" event={"ID":"e11ea02d-ac52-4318-9fdd-95c0bb171aed","Type":"ContainerStarted","Data":"0c9e32359e3ef256b0e8b244ad8f92a56f410da75819b00fd547aa422d729cb4"}
Nov 03 01:51:03.436156 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.436135    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd" event={"ID":"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd","Type":"ContainerStarted","Data":"68c010f0f76f47a515f05930ada535e464568c6ac669fdd14e110b17f7a77c3a"}
Nov 03 01:51:03.437474 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.437454    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-l87nq" event={"ID":"38de413a-873a-4288-85f5-c453581a53d6","Type":"ContainerStarted","Data":"566782c44891e8843e813f9d997fb62b2ea15f5ce5e8063a57f7789ffd20f296"}
Nov 03 01:51:03.437559 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.437489    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-l87nq" event={"ID":"38de413a-873a-4288-85f5-c453581a53d6","Type":"ContainerStarted","Data":"ee5c4e58b1d36c1d73804b6b742b1354d358c2693d05893eae4ee53d65b508c3"}
Nov 03 01:51:03.438535 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.438514    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-f5dm4" event={"ID":"48604fe3-9728-461e-ae37-8208b424e149","Type":"ContainerStarted","Data":"16c546e51f74fc98b52aef4f1f105191f19b7af9ee68374cd6c570e88ee5b764"}
Nov 03 01:51:03.439481 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.439461    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-gqz7v" event={"ID":"354d7018-887a-497f-b74f-195c8c88e184","Type":"ContainerStarted","Data":"3c243069849fdbd584653523b58a37178540579d86c55ed2731bcfe14e30bffe"}
Nov 03 01:51:03.440441 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.440423    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7t99k" event={"ID":"b25bd7be-290a-499a-a81d-c005793eace2","Type":"ContainerStarted","Data":"5ecfcc08a256f4626dbbeccb2c8ed8bb53b8fde82e97487eb2f786119c8230c2"}
Nov 03 01:51:03.441362 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.441344    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z28hp" event={"ID":"665f9d02-067e-4506-9793-cc7f17eb95bb","Type":"ContainerStarted","Data":"1d56f450ef97694ee4750e860271e17224a4d8210b40a24273628c3907ad8860"}
Nov 03 01:51:03.534522 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.534325    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:51:03.564748 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.564670    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mxg52"]
Nov 03 01:51:03.569928 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:51:03.569906    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79fe02fa_99ce_4222_b569_6b323df47ab4.slice/crio-47b4e6f25f392ebd934f9fa89d07ddb5b2ffa976c3422bc60f8c4ddaca5397ad WatchSource:0}: Error finding container 47b4e6f25f392ebd934f9fa89d07ddb5b2ffa976c3422bc60f8c4ddaca5397ad: Status 404 returned error can't find the container with id 47b4e6f25f392ebd934f9fa89d07ddb5b2ffa976c3422bc60f8c4ddaca5397ad
Nov 03 01:51:03.920087 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:03.920060    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mdr8p"]
Nov 03 01:51:03.923635 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:51:03.923598    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a56b4b_283d_4e63_bb6a_59c5d5df0c2c.slice/crio-99c457ae3480cd320c269741628c5969c011a89322c106e507d3670364521f08 WatchSource:0}: Error finding container 99c457ae3480cd320c269741628c5969c011a89322c106e507d3670364521f08: Status 404 returned error can't find the container with id 99c457ae3480cd320c269741628c5969c011a89322c106e507d3670364521f08
Nov 03 01:51:04.038634 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:04.038605    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:04.038823 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:04.038768    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 01:51:04.038823 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:04.038781    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bcfc5667d-w8v7m: secret "image-registry-tls" not found
Nov 03 01:51:04.038902 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:04.038828    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls podName:108d9224-ccd3-4925-a65b-0d59e938ff72 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:06.038812094 +0000 UTC m=+36.326382882 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls") pod "image-registry-6bcfc5667d-w8v7m" (UID: "108d9224-ccd3-4925-a65b-0d59e938ff72") : secret "image-registry-tls" not found
Nov 03 01:51:04.139369 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:04.139339    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:04.139578 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:04.139376    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-55bnn\" (UID: \"0110b28b-ec04-4f65-bc3d-2e82f88b30d7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn"
Nov 03 01:51:04.139578 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:04.139403    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:04.139578 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:04.139457    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qtpnh\" (UID: \"ef90f36a-279e-4bfa-a97d-70132a266670\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:04.139578 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:04.139489    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 01:51:04.139578 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:04.139513    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 01:51:04.139578 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:04.139493    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle podName:7a9e57fd-a6e6-4ddb-aa08-ac1390403544 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:06.139478192 +0000 UTC m=+36.427048970 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle") pod "router-default-7875dd7849-ptjkr" (UID: "7a9e57fd-a6e6-4ddb-aa08-ac1390403544") : configmap references non-existent config key: service-ca.crt
Nov 03 01:51:04.139578 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:04.139545    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 01:51:04.139578 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:04.139565    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert podName:0110b28b-ec04-4f65-bc3d-2e82f88b30d7 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:06.139549133 +0000 UTC m=+36.427119917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-55bnn" (UID: "0110b28b-ec04-4f65-bc3d-2e82f88b30d7") : secret "networking-console-plugin-cert" not found
Nov 03 01:51:04.139873 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:04.139588    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls podName:ef90f36a-279e-4bfa-a97d-70132a266670 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:06.139578815 +0000 UTC m=+36.427149596 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qtpnh" (UID: "ef90f36a-279e-4bfa-a97d-70132a266670") : secret "samples-operator-tls" not found
Nov 03 01:51:04.139873 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:04.139613    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs podName:7a9e57fd-a6e6-4ddb-aa08-ac1390403544 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:06.139608477 +0000 UTC m=+36.427179250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs") pod "router-default-7875dd7849-ptjkr" (UID: "7a9e57fd-a6e6-4ddb-aa08-ac1390403544") : secret "router-metrics-certs-default" not found
Nov 03 01:51:04.240357 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:04.240308    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:04.240482 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:04.240393    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:04.240482 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:04.240432    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 01:51:04.240482 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:04.240448    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55fd5cff67-bb76l: secret "image-registry-tls" not found
Nov 03 01:51:04.240601 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:04.240485    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls podName:9c546cea-b474-44cf-84a5-b89784bda1d6 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:06.240474252 +0000 UTC m=+36.528045025 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls") pod "image-registry-55fd5cff67-bb76l" (UID: "9c546cea-b474-44cf-84a5-b89784bda1d6") : secret "image-registry-tls" not found
Nov 03 01:51:04.240601 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:04.240484    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 01:51:04.240601 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:04.240532    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls podName:f4425c17-5cc6-4afc-922c-d74694805196 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:06.240517671 +0000 UTC m=+36.528088460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-g8wz8" (UID: "f4425c17-5cc6-4afc-922c-d74694805196") : secret "cluster-monitoring-operator-tls" not found
Nov 03 01:51:04.297858 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:04.297815    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-rcv8q" podStartSLOduration=446.297802102 podStartE2EDuration="7m26.297802102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:04.297416107 +0000 UTC m=+34.584986902" watchObservedRunningTime="2026-11-03 01:51:04.297802102 +0000 UTC m=+34.585372897"
Nov 03 01:51:04.341628 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:04.341601    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:04.341837 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:04.341714    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 01:51:04.341837 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:04.341766    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls podName:2555270f-a2ea-4d11-8dff-d130f9786e3d nodeName:}" failed. No retries permitted until 2026-11-03 01:51:06.341751873 +0000 UTC m=+36.629322647 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls") pod "dns-default-w4h82" (UID: "2555270f-a2ea-4d11-8dff-d130f9786e3d") : secret "dns-default-metrics-tls" not found
Nov 03 01:51:04.446032 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:04.446006    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mdr8p" event={"ID":"42a56b4b-283d-4e63-bb6a-59c5d5df0c2c","Type":"ContainerStarted","Data":"99c457ae3480cd320c269741628c5969c011a89322c106e507d3670364521f08"}
Nov 03 01:51:04.447354 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:04.447298    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mxg52" event={"ID":"79fe02fa-99ce-4222-b569-6b323df47ab4","Type":"ContainerStarted","Data":"47b4e6f25f392ebd934f9fa89d07ddb5b2ffa976c3422bc60f8c4ddaca5397ad"}
Nov 03 01:51:04.522767 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:04.522714    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-jqkfr" podStartSLOduration=446.522680834 podStartE2EDuration="7m26.522680834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:04.409817083 +0000 UTC m=+34.697387901" watchObservedRunningTime="2026-11-03 01:51:04.522680834 +0000 UTC m=+34.810251630"
Nov 03 01:51:06.061369 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:06.061324    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:06.061871 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:06.061648    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 01:51:06.061871 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:06.061668    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bcfc5667d-w8v7m: secret "image-registry-tls" not found
Nov 03 01:51:06.061871 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:06.061841    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls podName:108d9224-ccd3-4925-a65b-0d59e938ff72 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:10.061819936 +0000 UTC m=+40.349390724 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls") pod "image-registry-6bcfc5667d-w8v7m" (UID: "108d9224-ccd3-4925-a65b-0d59e938ff72") : secret "image-registry-tls" not found
Nov 03 01:51:06.163430 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:06.162887    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:06.163430 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:06.162942    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-55bnn\" (UID: \"0110b28b-ec04-4f65-bc3d-2e82f88b30d7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn"
Nov 03 01:51:06.163430 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:06.163046    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 01:51:06.163430 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:06.163069    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle podName:7a9e57fd-a6e6-4ddb-aa08-ac1390403544 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:10.163046822 +0000 UTC m=+40.450617617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle") pod "router-default-7875dd7849-ptjkr" (UID: "7a9e57fd-a6e6-4ddb-aa08-ac1390403544") : configmap references non-existent config key: service-ca.crt
Nov 03 01:51:06.163430 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:06.163099    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:06.163430 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:06.163157    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qtpnh\" (UID: \"ef90f36a-279e-4bfa-a97d-70132a266670\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:06.163430 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:06.163260    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 01:51:06.163430 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:06.163298    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls podName:ef90f36a-279e-4bfa-a97d-70132a266670 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:10.163283882 +0000 UTC m=+40.450854675 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qtpnh" (UID: "ef90f36a-279e-4bfa-a97d-70132a266670") : secret "samples-operator-tls" not found
Nov 03 01:51:06.163430 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:06.163350    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 01:51:06.163430 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:06.163392    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs podName:7a9e57fd-a6e6-4ddb-aa08-ac1390403544 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:10.163370789 +0000 UTC m=+40.450941594 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs") pod "router-default-7875dd7849-ptjkr" (UID: "7a9e57fd-a6e6-4ddb-aa08-ac1390403544") : secret "router-metrics-certs-default" not found
Nov 03 01:51:06.163430 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:06.163410    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert podName:0110b28b-ec04-4f65-bc3d-2e82f88b30d7 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:10.163400659 +0000 UTC m=+40.450971438 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-55bnn" (UID: "0110b28b-ec04-4f65-bc3d-2e82f88b30d7") : secret "networking-console-plugin-cert" not found
Nov 03 01:51:06.264035 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:06.263996    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:06.264207 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:06.264080    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:06.264297 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:06.264279    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 01:51:06.264360 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:06.264335    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls podName:f4425c17-5cc6-4afc-922c-d74694805196 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:10.264321812 +0000 UTC m=+40.551892585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-g8wz8" (UID: "f4425c17-5cc6-4afc-922c-d74694805196") : secret "cluster-monitoring-operator-tls" not found
Nov 03 01:51:06.264435 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:06.264380    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 01:51:06.264435 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:06.264392    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55fd5cff67-bb76l: secret "image-registry-tls" not found
Nov 03 01:51:06.264435 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:06.264426    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls podName:9c546cea-b474-44cf-84a5-b89784bda1d6 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:10.264414574 +0000 UTC m=+40.551985362 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls") pod "image-registry-55fd5cff67-bb76l" (UID: "9c546cea-b474-44cf-84a5-b89784bda1d6") : secret "image-registry-tls" not found
Nov 03 01:51:06.365518 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:06.365110    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:06.365518 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:06.365317    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 01:51:06.365518 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:06.365379    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls podName:2555270f-a2ea-4d11-8dff-d130f9786e3d nodeName:}" failed. No retries permitted until 2026-11-03 01:51:10.365360802 +0000 UTC m=+40.652931582 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls") pod "dns-default-w4h82" (UID: "2555270f-a2ea-4d11-8dff-d130f9786e3d") : secret "dns-default-metrics-tls" not found
Nov 03 01:51:07.507057 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:07.507014    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-f5dm4" event={"ID":"48604fe3-9728-461e-ae37-8208b424e149","Type":"ContainerStarted","Data":"78c22d45c6d5c52616b35fc2409a27c719cd0fa168d546eb937804770b61e1a1"}
Nov 03 01:51:07.623552 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:07.622342    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-l87nq" podStartSLOduration=449.622323731 podStartE2EDuration="7m29.622323731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:04.524513717 +0000 UTC m=+34.812084513" watchObservedRunningTime="2026-11-03 01:51:07.622323731 +0000 UTC m=+37.909894528"
Nov 03 01:51:07.623552 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:07.623352    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-f5dm4" podStartSLOduration=384.275307106 podStartE2EDuration="6m27.623341288s" totalImagesPullingTime="3.348034182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:44:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:07.62140257 +0000 UTC m=+37.908973365" watchObservedRunningTime="2026-11-03 01:51:07.623341288 +0000 UTC m=+37.910912082"
Nov 03 01:51:09.516850 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:09.516154    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-gqz7v" event={"ID":"354d7018-887a-497f-b74f-195c8c88e184","Type":"ContainerStarted","Data":"463d876c65e5e5075dfd59f6b8c35ea5f48d96b38817b9dfa7b504c63b6ba219"}
Nov 03 01:51:09.519293 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:09.518813    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn" event={"ID":"15face97-689e-4fef-86c0-17a2a85829db","Type":"ContainerStarted","Data":"e07ebe8dc2dfdac20310e43c2ebbb4d7ff208cbac7997abc252073c579cd57fd"}
Nov 03 01:51:09.631038 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:09.630969    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-gqz7v" podStartSLOduration=387.539372693 podStartE2EDuration="6m33.630949707s" totalImagesPullingTime="6.091577014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:09.630913351 +0000 UTC m=+39.918484148" watchObservedRunningTime="2026-11-03 01:51:09.630949707 +0000 UTC m=+39.918520504"
Nov 03 01:51:10.104450 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.104367    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:10.104607 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:10.104534    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 01:51:10.104607 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:10.104558    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bcfc5667d-w8v7m: secret "image-registry-tls" not found
Nov 03 01:51:10.104726 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:10.104621    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls podName:108d9224-ccd3-4925-a65b-0d59e938ff72 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:18.104599779 +0000 UTC m=+48.392170564 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls") pod "image-registry-6bcfc5667d-w8v7m" (UID: "108d9224-ccd3-4925-a65b-0d59e938ff72") : secret "image-registry-tls" not found
Nov 03 01:51:10.205090 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.205049    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:10.205284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.205113    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-55bnn\" (UID: \"0110b28b-ec04-4f65-bc3d-2e82f88b30d7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn"
Nov 03 01:51:10.205284 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.205207    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:10.205284 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:10.205245    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle podName:7a9e57fd-a6e6-4ddb-aa08-ac1390403544 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:18.20521562 +0000 UTC m=+48.492786415 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle") pod "router-default-7875dd7849-ptjkr" (UID: "7a9e57fd-a6e6-4ddb-aa08-ac1390403544") : configmap references non-existent config key: service-ca.crt
Nov 03 01:51:10.205284 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:10.205283    2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 01:51:10.205514 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:10.205315    2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 01:51:10.205514 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:10.205333    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert podName:0110b28b-ec04-4f65-bc3d-2e82f88b30d7 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:18.205321342 +0000 UTC m=+48.492892128 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-55bnn" (UID: "0110b28b-ec04-4f65-bc3d-2e82f88b30d7") : secret "networking-console-plugin-cert" not found
Nov 03 01:51:10.205514 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.205357    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qtpnh\" (UID: \"ef90f36a-279e-4bfa-a97d-70132a266670\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:10.205514 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:10.205397    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs podName:7a9e57fd-a6e6-4ddb-aa08-ac1390403544 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:18.205377144 +0000 UTC m=+48.492947936 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs") pod "router-default-7875dd7849-ptjkr" (UID: "7a9e57fd-a6e6-4ddb-aa08-ac1390403544") : secret "router-metrics-certs-default" not found
Nov 03 01:51:10.205514 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:10.205433    2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 01:51:10.205514 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:10.205465    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls podName:ef90f36a-279e-4bfa-a97d-70132a266670 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:18.205451412 +0000 UTC m=+48.493022186 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qtpnh" (UID: "ef90f36a-279e-4bfa-a97d-70132a266670") : secret "samples-operator-tls" not found
Nov 03 01:51:10.308350 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.307956    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:10.308350 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.308067    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:10.308350 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:10.308126    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 01:51:10.308350 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:10.308207    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls podName:f4425c17-5cc6-4afc-922c-d74694805196 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:18.308185891 +0000 UTC m=+48.595756678 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-g8wz8" (UID: "f4425c17-5cc6-4afc-922c-d74694805196") : secret "cluster-monitoring-operator-tls" not found
Nov 03 01:51:10.308350 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:10.308242    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 01:51:10.308350 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:10.308256    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55fd5cff67-bb76l: secret "image-registry-tls" not found
Nov 03 01:51:10.308350 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:10.308321    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls podName:9c546cea-b474-44cf-84a5-b89784bda1d6 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:18.308306664 +0000 UTC m=+48.595877455 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls") pod "image-registry-55fd5cff67-bb76l" (UID: "9c546cea-b474-44cf-84a5-b89784bda1d6") : secret "image-registry-tls" not found
Nov 03 01:51:10.409153 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.409109    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:10.409338 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:10.409259    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 01:51:10.409338 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:10.409303    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls podName:2555270f-a2ea-4d11-8dff-d130f9786e3d nodeName:}" failed. No retries permitted until 2026-11-03 01:51:18.409291818 +0000 UTC m=+48.696862595 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls") pod "dns-default-w4h82" (UID: "2555270f-a2ea-4d11-8dff-d130f9786e3d") : secret "dns-default-metrics-tls" not found
Nov 03 01:51:10.526412 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.526381    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/0.log"
Nov 03 01:51:10.526917 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.526427    2571 generic.go:356] "Generic (PLEG): container finished" podID="a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd" containerID="7ac872f18d4fddd45f91ff8fa50fac11476035b317d2add6df12227301b36a75" exitCode=255
Nov 03 01:51:10.526917 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.526764    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd" event={"ID":"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd","Type":"ContainerDied","Data":"7ac872f18d4fddd45f91ff8fa50fac11476035b317d2add6df12227301b36a75"}
Nov 03 01:51:10.526917 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.526887    2571 scope.go:122] "RemoveContainer" containerID="7ac872f18d4fddd45f91ff8fa50fac11476035b317d2add6df12227301b36a75"
Nov 03 01:51:10.528675 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.528651    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7t99k" event={"ID":"b25bd7be-290a-499a-a81d-c005793eace2","Type":"ContainerStarted","Data":"d9410f9222aac4a5ee5a82edd640827567420361b46f82d3474548a58b75ed04"}
Nov 03 01:51:10.531538 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.531510    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr" event={"ID":"baf5281f-5938-4417-b22f-2644034da7ef","Type":"ContainerStarted","Data":"7ecc92e2b116eace93766511ee1bfb6ebdf024d8dba845a6151a2b2fbd3a92e5"}
Nov 03 01:51:10.605688 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.605642    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn" podStartSLOduration=382.418788407 podStartE2EDuration="6m28.605628419s" totalImagesPullingTime="6.186840012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:09.747616869 +0000 UTC m=+40.035187665" watchObservedRunningTime="2026-11-03 01:51:10.605628419 +0000 UTC m=+40.893199214"
Nov 03 01:51:10.606790 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.606735    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9twrb"]
Nov 03 01:51:10.612142 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.611421    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:10.666041 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.665446    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9twrb"]
Nov 03 01:51:10.666041 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.665789    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jwnvd\""
Nov 03 01:51:10.666041 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.665845    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 01:51:10.666041 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.665884    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 01:51:10.711211 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.711175    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z56p\" (UniqueName: \"kubernetes.io/projected/be449663-2797-487f-8559-0a96c563ae3a-kube-api-access-5z56p\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:10.711211 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.711264    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:10.711211 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.711295    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be449663-2797-487f-8559-0a96c563ae3a-data-volume\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:10.711211 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.711336    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be449663-2797-487f-8559-0a96c563ae3a-crio-socket\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:10.711913 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.711890    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be449663-2797-487f-8559-0a96c563ae3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:10.757382 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.757301    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr" podStartSLOduration=382.071068354 podStartE2EDuration="6m28.757286211s" totalImagesPullingTime="6.686217857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:10.757149704 +0000 UTC m=+41.044720501" watchObservedRunningTime="2026-11-03 01:51:10.757286211 +0000 UTC m=+41.044857010"
Nov 03 01:51:10.815632 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.813612    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z56p\" (UniqueName: \"kubernetes.io/projected/be449663-2797-487f-8559-0a96c563ae3a-kube-api-access-5z56p\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:10.815632 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.813686    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:10.815632 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.813740    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be449663-2797-487f-8559-0a96c563ae3a-data-volume\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:10.815632 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.813775    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be449663-2797-487f-8559-0a96c563ae3a-crio-socket\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:10.815632 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.813817    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be449663-2797-487f-8559-0a96c563ae3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:10.815632 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:10.814003    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 01:51:10.815632 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:10.814066    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be449663-2797-487f-8559-0a96c563ae3a-kube-rbac-proxy-cm podName:be449663-2797-487f-8559-0a96c563ae3a nodeName:}" failed. No retries permitted until 2026-11-03 01:51:11.314047463 +0000 UTC m=+41.601618254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/be449663-2797-487f-8559-0a96c563ae3a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9twrb" (UID: "be449663-2797-487f-8559-0a96c563ae3a") : configmap "kube-rbac-proxy" not found
Nov 03 01:51:10.815632 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:10.815133    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 01:51:10.815632 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:10.815187    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls podName:be449663-2797-487f-8559-0a96c563ae3a nodeName:}" failed. No retries permitted until 2026-11-03 01:51:11.315171763 +0000 UTC m=+41.602742553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9twrb" (UID: "be449663-2797-487f-8559-0a96c563ae3a") : secret "insights-runtime-extractor-tls" not found
Nov 03 01:51:10.815632 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.815506    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be449663-2797-487f-8559-0a96c563ae3a-data-volume\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:10.815632 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.815585    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be449663-2797-487f-8559-0a96c563ae3a-crio-socket\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:10.861301 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.861222    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-gxg2j-p5824"]
Nov 03 01:51:10.864812 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.864783    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:51:10.875177 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.875026    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z56p\" (UniqueName: \"kubernetes.io/projected/be449663-2797-487f-8559-0a96c563ae3a-kube-api-access-5z56p\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:10.898875 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.898841    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-t9rqp"]
Nov 03 01:51:10.904416 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.904390    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t9rqp"
Nov 03 01:51:10.911937 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.911909    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4wq24"]
Nov 03 01:51:10.915281 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.915255    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4wq24"
Nov 03 01:51:10.915532 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.915477    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-gxg2j-p5824"]
Nov 03 01:51:10.956836 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.956802    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-t9rqp"]
Nov 03 01:51:10.969915 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:10.969848    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4wq24"]
Nov 03 01:51:11.002486 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.002427    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7t99k" podStartSLOduration=2.123412731 podStartE2EDuration="9.002409875s" totalImagesPullingTime="6.878997144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:11.002022854 +0000 UTC m=+41.289593660" watchObservedRunningTime="2026-11-03 01:51:11.002409875 +0000 UTC m=+41.289980675"
Nov 03 01:51:11.015248 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.015208    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z46t7\" (UniqueName: \"kubernetes.io/projected/62c3ca06-b2a4-49d7-84db-1390277be2c8-kube-api-access-z46t7\") pod \"periodic-gathering-gxg2j-p5824\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") " pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:51:11.015411 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.015259    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f76x5\" (UniqueName: \"kubernetes.io/projected/cb8ca898-6f5f-4727-9d6d-d6aed65adfae-kube-api-access-f76x5\") pod \"migrator-66df5d496b-4wq24\" (UID: \"cb8ca898-6f5f-4727-9d6d-d6aed65adfae\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4wq24"
Nov 03 01:51:11.015411 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.015332    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62c3ca06-b2a4-49d7-84db-1390277be2c8-trusted-ca-bundle\") pod \"periodic-gathering-gxg2j-p5824\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") " pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:51:11.015411 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.015361    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62c3ca06-b2a4-49d7-84db-1390277be2c8-service-ca-bundle\") pod \"periodic-gathering-gxg2j-p5824\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") " pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:51:11.015588 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.015467    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/62c3ca06-b2a4-49d7-84db-1390277be2c8-archives-path\") pod \"periodic-gathering-gxg2j-p5824\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") " pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:51:11.015588 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.015491    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5jp6\" (UniqueName: \"kubernetes.io/projected/62150808-f089-4723-aac5-d17d3a803abe-kube-api-access-g5jp6\") pod \"migrator-66df5d496b-t9rqp\" (UID: \"62150808-f089-4723-aac5-d17d3a803abe\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t9rqp"
Nov 03 01:51:11.019813 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.019780    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-d7skx\""
Nov 03 01:51:11.020055 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.020032    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 01:51:11.020055 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.020032    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 01:51:11.116571 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.116491    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62c3ca06-b2a4-49d7-84db-1390277be2c8-service-ca-bundle\") pod \"periodic-gathering-gxg2j-p5824\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") " pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:51:11.116776 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.116584    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/62c3ca06-b2a4-49d7-84db-1390277be2c8-archives-path\") pod \"periodic-gathering-gxg2j-p5824\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") " pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:51:11.116776 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.116723    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5jp6\" (UniqueName: \"kubernetes.io/projected/62150808-f089-4723-aac5-d17d3a803abe-kube-api-access-g5jp6\") pod \"migrator-66df5d496b-t9rqp\" (UID: \"62150808-f089-4723-aac5-d17d3a803abe\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t9rqp"
Nov 03 01:51:11.117174 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.117137    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/62c3ca06-b2a4-49d7-84db-1390277be2c8-archives-path\") pod \"periodic-gathering-gxg2j-p5824\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") " pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:51:11.117299 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.117275    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62c3ca06-b2a4-49d7-84db-1390277be2c8-service-ca-bundle\") pod \"periodic-gathering-gxg2j-p5824\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") " pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:51:11.119986 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.119953    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z46t7\" (UniqueName: \"kubernetes.io/projected/62c3ca06-b2a4-49d7-84db-1390277be2c8-kube-api-access-z46t7\") pod \"periodic-gathering-gxg2j-p5824\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") " pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:51:11.120098 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.119997    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f76x5\" (UniqueName: \"kubernetes.io/projected/cb8ca898-6f5f-4727-9d6d-d6aed65adfae-kube-api-access-f76x5\") pod \"migrator-66df5d496b-4wq24\" (UID: \"cb8ca898-6f5f-4727-9d6d-d6aed65adfae\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4wq24"
Nov 03 01:51:11.120098 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.120072    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62c3ca06-b2a4-49d7-84db-1390277be2c8-trusted-ca-bundle\") pod \"periodic-gathering-gxg2j-p5824\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") " pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:51:11.121001 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.120957    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62c3ca06-b2a4-49d7-84db-1390277be2c8-trusted-ca-bundle\") pod \"periodic-gathering-gxg2j-p5824\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") " pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:51:11.175448 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.175416    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5jp6\" (UniqueName: \"kubernetes.io/projected/62150808-f089-4723-aac5-d17d3a803abe-kube-api-access-g5jp6\") pod \"migrator-66df5d496b-t9rqp\" (UID: \"62150808-f089-4723-aac5-d17d3a803abe\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t9rqp"
Nov 03 01:51:11.182980 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.182954    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f76x5\" (UniqueName: \"kubernetes.io/projected/cb8ca898-6f5f-4727-9d6d-d6aed65adfae-kube-api-access-f76x5\") pod \"migrator-66df5d496b-4wq24\" (UID: \"cb8ca898-6f5f-4727-9d6d-d6aed65adfae\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4wq24"
Nov 03 01:51:11.183114 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.183027    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z46t7\" (UniqueName: \"kubernetes.io/projected/62c3ca06-b2a4-49d7-84db-1390277be2c8-kube-api-access-z46t7\") pod \"periodic-gathering-gxg2j-p5824\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") " pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:51:11.193816 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.193790    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:51:11.221550 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.221516    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t9rqp"
Nov 03 01:51:11.232416 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.232383    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4wq24"
Nov 03 01:51:11.326370 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.326197    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be449663-2797-487f-8559-0a96c563ae3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:11.326498 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.326452    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:11.326682 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:11.326600    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 01:51:11.326682 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:11.326663    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls podName:be449663-2797-487f-8559-0a96c563ae3a nodeName:}" failed. No retries permitted until 2026-11-03 01:51:12.326644337 +0000 UTC m=+42.614215125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9twrb" (UID: "be449663-2797-487f-8559-0a96c563ae3a") : secret "insights-runtime-extractor-tls" not found
Nov 03 01:51:11.326840 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:11.326728    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 01:51:11.326840 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:11.326762    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be449663-2797-487f-8559-0a96c563ae3a-kube-rbac-proxy-cm podName:be449663-2797-487f-8559-0a96c563ae3a nodeName:}" failed. No retries permitted until 2026-11-03 01:51:12.3267509 +0000 UTC m=+42.614321686 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/be449663-2797-487f-8559-0a96c563ae3a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9twrb" (UID: "be449663-2797-487f-8559-0a96c563ae3a") : configmap "kube-rbac-proxy" not found
Nov 03 01:51:11.536268 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.536230    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mdr8p" event={"ID":"42a56b4b-283d-4e63-bb6a-59c5d5df0c2c","Type":"ContainerStarted","Data":"6b31b67c3a692509b6ae0f470a8829a78658c2c8d80d198cd9dda18d92880733"}
Nov 03 01:51:11.537164 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.536332    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:51:11.538133 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.538088    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z28hp" event={"ID":"665f9d02-067e-4506-9793-cc7f17eb95bb","Type":"ContainerStarted","Data":"e327fe8c8de4189b13fede0d8597642561fb73443b8d1fc1631941e04fde40a3"}
Nov 03 01:51:11.539789 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.539766    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/1.log"
Nov 03 01:51:11.540268 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.540247    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/0.log"
Nov 03 01:51:11.540901 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.540301    2571 generic.go:356] "Generic (PLEG): container finished" podID="a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd" containerID="8df6474c221450b3bf6a0de2c5673ab2c142dbf0053634b6d05d2b7c199801e8" exitCode=255
Nov 03 01:51:11.540901 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.540386    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd" event={"ID":"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd","Type":"ContainerDied","Data":"8df6474c221450b3bf6a0de2c5673ab2c142dbf0053634b6d05d2b7c199801e8"}
Nov 03 01:51:11.540901 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.540428    2571 scope.go:122] "RemoveContainer" containerID="7ac872f18d4fddd45f91ff8fa50fac11476035b317d2add6df12227301b36a75"
Nov 03 01:51:11.540901 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.540646    2571 scope.go:122] "RemoveContainer" containerID="8df6474c221450b3bf6a0de2c5673ab2c142dbf0053634b6d05d2b7c199801e8"
Nov 03 01:51:11.540901 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:11.540851    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rjmbd_openshift-console-operator(a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd" podUID="a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd"
Nov 03 01:51:11.583491 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.583459    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-gxg2j-p5824"]
Nov 03 01:51:11.586873 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:51:11.586838    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c3ca06_b2a4_49d7_84db_1390277be2c8.slice/crio-e59b0b9e967622046c45b1bbd1249b6ea96e8186fc6340e94dd34ee27174e7ac WatchSource:0}: Error finding container e59b0b9e967622046c45b1bbd1249b6ea96e8186fc6340e94dd34ee27174e7ac: Status 404 returned error can't find the container with id e59b0b9e967622046c45b1bbd1249b6ea96e8186fc6340e94dd34ee27174e7ac
Nov 03 01:51:11.633593 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.633564    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-t9rqp"]
Nov 03 01:51:11.637006 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:51:11.636982    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62150808_f089_4723_aac5_d17d3a803abe.slice/crio-1b1cabad9b401aeae3bd442116212c87000eae80b7baad5c9ac7d4cf503832f3 WatchSource:0}: Error finding container 1b1cabad9b401aeae3bd442116212c87000eae80b7baad5c9ac7d4cf503832f3: Status 404 returned error can't find the container with id 1b1cabad9b401aeae3bd442116212c87000eae80b7baad5c9ac7d4cf503832f3
Nov 03 01:51:11.649616 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.649566    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mdr8p" podStartSLOduration=33.372324917 podStartE2EDuration="40.649549443s" totalImagesPullingTime="7.277224526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:11.648334895 +0000 UTC m=+41.935905689" watchObservedRunningTime="2026-11-03 01:51:11.649549443 +0000 UTC m=+41.937120238"
Nov 03 01:51:11.683928 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.683863    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4wq24"]
Nov 03 01:51:11.687097 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:51:11.687071    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8ca898_6f5f_4727_9d6d_d6aed65adfae.slice/crio-6c6141e7f39691eba8f6633218ff32e37cd22cc731def6a22005e4618c1fb3be WatchSource:0}: Error finding container 6c6141e7f39691eba8f6633218ff32e37cd22cc731def6a22005e4618c1fb3be: Status 404 returned error can't find the container with id 6c6141e7f39691eba8f6633218ff32e37cd22cc731def6a22005e4618c1fb3be
Nov 03 01:51:11.760535 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:11.760476    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z28hp" podStartSLOduration=387.496684896 podStartE2EDuration="6m34.760460133s" totalImagesPullingTime="7.263775237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:11.760416788 +0000 UTC m=+42.047987581" watchObservedRunningTime="2026-11-03 01:51:11.760460133 +0000 UTC m=+42.048030955"
Nov 03 01:51:12.168170 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.166774    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-qlt9s"]
Nov 03 01:51:12.170670 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.170650    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-qlt9s"
Nov 03 01:51:12.222639 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.222607    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-qlt9s"]
Nov 03 01:51:12.223728 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.223632    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 01:51:12.223728 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.223653    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 01:51:12.223728 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.223653    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 01:51:12.223947 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.223736    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 01:51:12.223947 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.223813    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-h8v6w\""
Nov 03 01:51:12.236198 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.236174    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d0de4046-bfed-4695-af0e-1ba806e6b031-signing-key\") pod \"service-ca-5fffc97bdf-qlt9s\" (UID: \"d0de4046-bfed-4695-af0e-1ba806e6b031\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlt9s"
Nov 03 01:51:12.236341 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.236326    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkrns\" (UniqueName: \"kubernetes.io/projected/d0de4046-bfed-4695-af0e-1ba806e6b031-kube-api-access-dkrns\") pod \"service-ca-5fffc97bdf-qlt9s\" (UID: \"d0de4046-bfed-4695-af0e-1ba806e6b031\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlt9s"
Nov 03 01:51:12.236417 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.236391    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d0de4046-bfed-4695-af0e-1ba806e6b031-signing-cabundle\") pod \"service-ca-5fffc97bdf-qlt9s\" (UID: \"d0de4046-bfed-4695-af0e-1ba806e6b031\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlt9s"
Nov 03 01:51:12.337407 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.337375    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:12.337574 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.337412    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkrns\" (UniqueName: \"kubernetes.io/projected/d0de4046-bfed-4695-af0e-1ba806e6b031-kube-api-access-dkrns\") pod \"service-ca-5fffc97bdf-qlt9s\" (UID: \"d0de4046-bfed-4695-af0e-1ba806e6b031\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlt9s"
Nov 03 01:51:12.337574 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.337478    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d0de4046-bfed-4695-af0e-1ba806e6b031-signing-cabundle\") pod \"service-ca-5fffc97bdf-qlt9s\" (UID: \"d0de4046-bfed-4695-af0e-1ba806e6b031\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlt9s"
Nov 03 01:51:12.337574 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.337505    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be449663-2797-487f-8559-0a96c563ae3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:12.337574 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:12.337535    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 01:51:12.337755 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.337580    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d0de4046-bfed-4695-af0e-1ba806e6b031-signing-key\") pod \"service-ca-5fffc97bdf-qlt9s\" (UID: \"d0de4046-bfed-4695-af0e-1ba806e6b031\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlt9s"
Nov 03 01:51:12.337755 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:12.337608    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls podName:be449663-2797-487f-8559-0a96c563ae3a nodeName:}" failed. No retries permitted until 2026-11-03 01:51:14.337588742 +0000 UTC m=+44.625159533 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9twrb" (UID: "be449663-2797-487f-8559-0a96c563ae3a") : secret "insights-runtime-extractor-tls" not found
Nov 03 01:51:12.338077 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:12.338053    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 01:51:12.338168 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:12.338103    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be449663-2797-487f-8559-0a96c563ae3a-kube-rbac-proxy-cm podName:be449663-2797-487f-8559-0a96c563ae3a nodeName:}" failed. No retries permitted until 2026-11-03 01:51:14.338091945 +0000 UTC m=+44.625662717 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/be449663-2797-487f-8559-0a96c563ae3a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9twrb" (UID: "be449663-2797-487f-8559-0a96c563ae3a") : configmap "kube-rbac-proxy" not found
Nov 03 01:51:12.338534 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.338512    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d0de4046-bfed-4695-af0e-1ba806e6b031-signing-cabundle\") pod \"service-ca-5fffc97bdf-qlt9s\" (UID: \"d0de4046-bfed-4695-af0e-1ba806e6b031\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlt9s"
Nov 03 01:51:12.340115 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.340090    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d0de4046-bfed-4695-af0e-1ba806e6b031-signing-key\") pod \"service-ca-5fffc97bdf-qlt9s\" (UID: \"d0de4046-bfed-4695-af0e-1ba806e6b031\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlt9s"
Nov 03 01:51:12.395237 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.395209    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkrns\" (UniqueName: \"kubernetes.io/projected/d0de4046-bfed-4695-af0e-1ba806e6b031-kube-api-access-dkrns\") pod \"service-ca-5fffc97bdf-qlt9s\" (UID: \"d0de4046-bfed-4695-af0e-1ba806e6b031\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlt9s"
Nov 03 01:51:12.484384 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.484349    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-qlt9s"
Nov 03 01:51:12.553202 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.553179    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/1.log"
Nov 03 01:51:12.553615 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.553599    2571 scope.go:122] "RemoveContainer" containerID="8df6474c221450b3bf6a0de2c5673ab2c142dbf0053634b6d05d2b7c199801e8"
Nov 03 01:51:12.553862 ip-10-0-139-26 kubenswrapper[2571]: E1103 01:51:12.553838    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rjmbd_openshift-console-operator(a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd" podUID="a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd"
Nov 03 01:51:12.555644 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.555620    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4wq24" event={"ID":"cb8ca898-6f5f-4727-9d6d-d6aed65adfae","Type":"ContainerStarted","Data":"6c6141e7f39691eba8f6633218ff32e37cd22cc731def6a22005e4618c1fb3be"}
Nov 03 01:51:12.556546 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.556527    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t9rqp" event={"ID":"62150808-f089-4723-aac5-d17d3a803abe","Type":"ContainerStarted","Data":"1b1cabad9b401aeae3bd442116212c87000eae80b7baad5c9ac7d4cf503832f3"}
Nov 03 01:51:12.559799 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.559218    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gxg2j-p5824" event={"ID":"62c3ca06-b2a4-49d7-84db-1390277be2c8","Type":"ContainerStarted","Data":"d3f053ab4b69e0277c6fa3e29443517b0e6657a53305ff725e86df346681d4eb"}
Nov 03 01:51:12.559799 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.559259    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gxg2j-p5824" event={"ID":"62c3ca06-b2a4-49d7-84db-1390277be2c8","Type":"ContainerStarted","Data":"e59b0b9e967622046c45b1bbd1249b6ea96e8186fc6340e94dd34ee27174e7ac"}
Nov 03 01:51:12.784343 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.784286    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-gxg2j-p5824" podStartSLOduration=2.784274121 podStartE2EDuration="2.784274121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:12.783132335 +0000 UTC m=+43.070703131" watchObservedRunningTime="2026-11-03 01:51:12.784274121 +0000 UTC m=+43.071844914"
Nov 03 01:51:12.853971 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.853937    2571 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:12.854115 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.854021    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:12.864225 ip-10-0-139-26 kubenswrapper[2571]: I1103 01:51:12.864194    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-qlt9s"]
Nov 03 01:51:12.868020 ip-10-0-139-26 kubenswrapper[2571]: W1103 01:51:12.867987    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0de4046_bfed_4695_af0e_1ba806e6b031.slice/crio-3b60f291a818eea4d3505e73812647b86256e017b112bfafc09a0e703be5fe16 WatchSource:0}: Error finding container 3b60f291a818eea4d3505e73812647b86256e017b112bfafc09a0e703be5fe16: Status 404 returned error can't find the container with id 3b60f291a818eea4d3505e73812647b86256e017b112bfafc09a0e703be5fe16
Nov 03 01:51:12.922099 ip-10-0-139-26 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 01:51:12.937016 ip-10-0-139-26 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 01:51:12.937173 ip-10-0-139-26 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 01:51:12.937223 ip-10-0-139-26 systemd[1]: kubelet.service: Consumed 3.050s CPU time.
Nov 03 01:51:12.953361 ip-10-0-139-26 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 01:51:13.014662 ip-10-0-139-26 kubenswrapper[5005]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 01:51:13.014662 ip-10-0-139-26 kubenswrapper[5005]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 01:51:13.014662 ip-10-0-139-26 kubenswrapper[5005]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 01:51:13.014662 ip-10-0-139-26 kubenswrapper[5005]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 01:51:13.016742 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016644    5005 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:51:13.016742 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016742    5005 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:51:13.016807 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016746    5005 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:51:13.016807 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016749    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:51:13.016807 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016752    5005 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:51:13.016807 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016755    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:51:13.016807 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016759    5005 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:51:13.016807 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016764    5005 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:51:13.016807 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016767    5005 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:51:13.016807 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016770    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:51:13.016807 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016773    5005 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:51:13.016807 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016776    5005 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:51:13.016807 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016779    5005 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:51:13.016807 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016788    5005 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:51:13.016807 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016791    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:51:13.016807 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016794    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:51:13.016807 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016797    5005 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:51:13.016807 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016800    5005 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:51:13.016807 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016802    5005 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:51:13.016807 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016805    5005 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:51:13.016807 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016807    5005 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:51:13.017296 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016810    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:51:13.017296 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016813    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:51:13.017296 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016815    5005 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:51:13.017296 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016818    5005 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:51:13.017296 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016821    5005 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:51:13.017296 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016823    5005 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:51:13.017296 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016826    5005 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:51:13.017296 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016829    5005 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:51:13.017296 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016832    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:51:13.017296 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016834    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:51:13.017296 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016837    5005 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:51:13.017296 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016839    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:51:13.017296 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016842    5005 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:51:13.017296 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016844    5005 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:51:13.017296 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016847    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:51:13.017296 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016849    5005 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:51:13.017296 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016853    5005 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:51:13.017296 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016857    5005 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:51:13.017296 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016860    5005 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:51:13.017787 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016862    5005 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:51:13.017787 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016865    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:51:13.017787 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016872    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:51:13.017787 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016875    5005 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:51:13.017787 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016877    5005 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:51:13.017787 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016880    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:51:13.017787 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016882    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:51:13.017787 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016885    5005 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:51:13.017787 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016887    5005 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:51:13.017787 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016890    5005 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:51:13.017787 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016893    5005 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:51:13.017787 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016897    5005 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:51:13.017787 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016900    5005 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:51:13.017787 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016902    5005 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:51:13.017787 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016905    5005 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:51:13.017787 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016907    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:51:13.017787 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016910    5005 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:51:13.017787 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016912    5005 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:51:13.017787 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016915    5005 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:51:13.017787 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016918    5005 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:51:13.018260 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016920    5005 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:51:13.018260 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016923    5005 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:51:13.018260 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016926    5005 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:51:13.018260 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016928    5005 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:51:13.018260 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016932    5005 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:51:13.018260 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016934    5005 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:51:13.018260 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016937    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:51:13.018260 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016939    5005 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:51:13.018260 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016942    5005 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:51:13.018260 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016945    5005 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:51:13.018260 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016947    5005 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:51:13.018260 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016950    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:51:13.018260 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016952    5005 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:51:13.018260 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016955    5005 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:51:13.018260 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016957    5005 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:51:13.018260 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016960    5005 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:51:13.018260 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016963    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:51:13.018260 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016966    5005 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:51:13.018260 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016968    5005 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:51:13.018732 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016971    5005 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:51:13.018732 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016973    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:51:13.018732 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016977    5005 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:51:13.018732 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016980    5005 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:51:13.018732 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016982    5005 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:51:13.018732 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016985    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:51:13.018732 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016987    5005 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:51:13.018732 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016990    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:51:13.018732 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016992    5005 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:51:13.018732 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016995    5005 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:51:13.018732 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.016998    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:51:13.018732 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017001    5005 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:51:13.018732 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017004    5005 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:51:13.018732 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017006    5005 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:51:13.018732 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017009    5005 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:51:13.018732 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017011    5005 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:51:13.018732 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017014    5005 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:51:13.018732 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017016    5005 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:51:13.018732 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017019    5005 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:51:13.018732 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017021    5005 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:51:13.019241 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017023    5005 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:51:13.019241 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017026    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:51:13.019241 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017029    5005 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:51:13.019241 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017031    5005 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:51:13.019241 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017034    5005 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:51:13.019241 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017036    5005 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:51:13.019241 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017039    5005 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:51:13.019241 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017763    5005 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:51:13.019241 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017771    5005 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:51:13.019241 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017775    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:51:13.019241 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017778    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:51:13.019241 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017781    5005 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:51:13.019241 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017784    5005 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:51:13.019241 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017787    5005 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:51:13.019241 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017791    5005 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:51:13.019241 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017796    5005 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:51:13.019241 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017800    5005 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:51:13.019241 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017804    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:51:13.019241 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017807    5005 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:51:13.019736 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017810    5005 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:51:13.019736 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017813    5005 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:51:13.019736 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017816    5005 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:51:13.019736 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017818    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:51:13.019736 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017821    5005 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:51:13.019736 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017823    5005 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:51:13.019736 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017826    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:51:13.019736 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017829    5005 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:51:13.019736 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017831    5005 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:51:13.019736 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017834    5005 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:51:13.019736 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017836    5005 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:51:13.019736 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017839    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:51:13.019736 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017841    5005 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:51:13.019736 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017844    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:51:13.019736 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017847    5005 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:51:13.019736 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017849    5005 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:51:13.019736 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017851    5005 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:51:13.019736 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017854    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:51:13.019736 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017856    5005 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:51:13.019736 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017859    5005 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:51:13.020243 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017861    5005 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:51:13.020243 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017864    5005 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:51:13.020243 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017867    5005 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:51:13.020243 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017876    5005 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:51:13.020243 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017879    5005 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:51:13.020243 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017882    5005 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:51:13.020243 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017885    5005 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:51:13.020243 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017887    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:51:13.020243 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017890    5005 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:51:13.020243 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017893    5005 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:51:13.020243 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017895    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:51:13.020243 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017898    5005 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:51:13.020243 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017900    5005 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:51:13.020243 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017903    5005 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:51:13.020243 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017905    5005 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:51:13.020243 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017908    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:51:13.020243 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017911    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:51:13.020243 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017913    5005 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:51:13.020243 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017916    5005 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:51:13.020700 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017918    5005 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:51:13.020700 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017920    5005 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:51:13.020700 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017923    5005 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:51:13.020700 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017925    5005 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:51:13.020700 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017928    5005 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:51:13.020700 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017931    5005 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:51:13.020700 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017933    5005 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:51:13.020700 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017936    5005 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:51:13.020700 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017939    5005 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:51:13.020700 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017941    5005 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:51:13.020700 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017947    5005 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:51:13.020700 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017949    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:51:13.020700 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017952    5005 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:51:13.020700 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017954    5005 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:51:13.020700 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017957    5005 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:51:13.020700 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017960    5005 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:51:13.020700 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017962    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:51:13.020700 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017966    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:51:13.020700 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017969    5005 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:51:13.020700 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017972    5005 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:51:13.021257 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017974    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:51:13.021257 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017976    5005 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:51:13.021257 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017979    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:51:13.021257 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017982    5005 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:51:13.021257 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017984    5005 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:51:13.021257 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017987    5005 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:51:13.021257 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017989    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:51:13.021257 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017992    5005 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:51:13.021257 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017994    5005 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:51:13.021257 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017997    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:51:13.021257 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.017999    5005 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:51:13.021257 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018002    5005 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:51:13.021257 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018004    5005 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:51:13.021257 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018007    5005 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:51:13.021257 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018009    5005 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:51:13.021257 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018012    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:51:13.021257 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018015    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:51:13.021257 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018018    5005 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:51:13.021257 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018021    5005 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:51:13.021257 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018023    5005 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:51:13.021754 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018026    5005 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:51:13.021754 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018028    5005 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:51:13.021754 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018032    5005 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:51:13.021754 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018035    5005 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:51:13.021754 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018038    5005 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:51:13.021754 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018041    5005 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:51:13.021754 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018044    5005 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:51:13.021754 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018046    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:51:13.021754 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018049    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:51:13.021754 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018051    5005 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:51:13.021754 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018054    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:51:13.021754 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018056    5005 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:51:13.021754 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018059    5005 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:51:13.021754 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018062    5005 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:51:13.021754 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018064    5005 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:51:13.021754 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018548    5005 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:51:13.021754 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018554    5005 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:51:13.021754 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018557    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:51:13.021754 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018560    5005 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:51:13.021754 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018562    5005 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:51:13.022232 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018565    5005 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:51:13.022232 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018568    5005 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:51:13.022232 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018571    5005 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:51:13.022232 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018574    5005 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:51:13.022232 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018576    5005 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:51:13.022232 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018579    5005 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:51:13.022232 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018581    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:51:13.022232 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018584    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:51:13.022232 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018587    5005 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:51:13.022232 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018590    5005 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:51:13.022232 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018593    5005 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:51:13.022232 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018595    5005 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:51:13.022232 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018598    5005 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:51:13.022232 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018600    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:51:13.022232 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018604    5005 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:51:13.022232 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018607    5005 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:51:13.022232 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018610    5005 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:51:13.022232 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018613    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:51:13.022232 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018615    5005 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:51:13.022232 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018618    5005 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:51:13.022782 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018621    5005 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:51:13.022782 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018623    5005 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:51:13.022782 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018626    5005 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:51:13.022782 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018629    5005 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:51:13.022782 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018632    5005 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:51:13.022782 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018634    5005 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:51:13.022782 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018637    5005 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:51:13.022782 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018640    5005 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:51:13.022782 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018642    5005 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:51:13.022782 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018645    5005 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:51:13.022782 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018647    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:51:13.022782 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018650    5005 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:51:13.022782 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018652    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:51:13.022782 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018655    5005 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:51:13.022782 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018657    5005 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:51:13.022782 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018660    5005 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:51:13.022782 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018663    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:51:13.022782 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018665    5005 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:51:13.022782 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018667    5005 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:51:13.022782 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018670    5005 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:51:13.023287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018673    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:51:13.023287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018675    5005 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:51:13.023287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018678    5005 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:51:13.023287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018681    5005 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:51:13.023287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018683    5005 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:51:13.023287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018686    5005 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:51:13.023287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018704    5005 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:51:13.023287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018708    5005 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:51:13.023287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018711    5005 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:51:13.023287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018714    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:51:13.023287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018716    5005 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:51:13.023287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018721    5005 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:51:13.023287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018725    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:51:13.023287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018728    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:51:13.023287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018731    5005 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:51:13.023287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018734    5005 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:51:13.023287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018737    5005 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:51:13.023287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018740    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:51:13.023287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018743    5005 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:51:13.023768 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018745    5005 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:51:13.023768 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018748    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:51:13.023768 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018750    5005 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:51:13.023768 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018753    5005 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:51:13.023768 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018755    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:51:13.023768 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018758    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:51:13.023768 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018761    5005 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:51:13.023768 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018763    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:51:13.023768 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018766    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:51:13.023768 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018769    5005 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:51:13.023768 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018771    5005 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:51:13.023768 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018774    5005 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:51:13.023768 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018776    5005 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:51:13.023768 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018779    5005 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:51:13.023768 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018782    5005 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:51:13.023768 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018784    5005 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:51:13.023768 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018787    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:51:13.023768 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018789    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:51:13.023768 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018792    5005 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:51:13.024274 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018796    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:51:13.024274 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018798    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:51:13.024274 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018801    5005 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:51:13.024274 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018803    5005 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:51:13.024274 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018806    5005 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:51:13.024274 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018811    5005 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:51:13.024274 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018815    5005 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:51:13.024274 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018818    5005 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:51:13.024274 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018821    5005 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:51:13.024274 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018824    5005 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:51:13.024274 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018827    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:51:13.024274 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018831    5005 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:51:13.024274 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018834    5005 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:51:13.024274 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018836    5005 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:51:13.024274 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018839    5005 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:51:13.024274 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018842    5005 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:51:13.024274 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018844    5005 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:51:13.024274 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018847    5005 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:51:13.024274 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018850    5005 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:51:13.024274 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018852    5005 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018855    5005 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018857    5005 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.018860    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.018968    5005 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.018975    5005 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.018981    5005 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.018986    5005 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.018993    5005 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.018996    5005 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019000    5005 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019004    5005 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019008    5005 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019012    5005 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019016    5005 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019019    5005 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019023    5005 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019026    5005 flags.go:64] FLAG: --cgroup-root=""
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019029    5005 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019032    5005 flags.go:64] FLAG: --client-ca-file=""
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019036    5005 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019039    5005 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019044    5005 flags.go:64] FLAG: --cluster-domain=""
Nov 03 01:51:13.025862 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019047    5005 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019050    5005 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019054    5005 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019058    5005 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019062    5005 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019065    5005 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019068    5005 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019071    5005 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019074    5005 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019078    5005 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019081    5005 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019084    5005 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019087    5005 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019092    5005 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019095    5005 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019098    5005 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019101    5005 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019105    5005 flags.go:64] FLAG: --enable-server="true"
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019110    5005 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019117    5005 flags.go:64] FLAG: --event-burst="100"
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019121    5005 flags.go:64] FLAG: --event-qps="50"
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019126    5005 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019132    5005 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019137    5005 flags.go:64] FLAG: --eviction-hard=""
Nov 03 01:51:13.026399 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019143    5005 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019148    5005 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019153    5005 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019158    5005 flags.go:64] FLAG: --eviction-soft=""
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019163    5005 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019167    5005 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019172    5005 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019177    5005 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019182    5005 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019186    5005 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019192    5005 flags.go:64] FLAG: --feature-gates=""
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019198    5005 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019202    5005 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019205    5005 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019208    5005 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019212    5005 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019215    5005 flags.go:64] FLAG: --help="false"
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019218    5005 flags.go:64] FLAG: --hostname-override="ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019221    5005 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019224    5005 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019227    5005 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019231    5005 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019234    5005 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 01:51:13.027019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019237    5005 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019240    5005 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019246    5005 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019249    5005 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019252    5005 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019257    5005 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019262    5005 flags.go:64] FLAG: --kube-reserved=""
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019269    5005 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019274    5005 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019278    5005 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019281    5005 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019284    5005 flags.go:64] FLAG: --lock-file=""
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019287    5005 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019290    5005 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019293    5005 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019299    5005 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019302    5005 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019305    5005 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019309    5005 flags.go:64] FLAG: --logging-format="text"
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019311    5005 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019316    5005 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019319    5005 flags.go:64] FLAG: --manifest-url=""
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019322    5005 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019326    5005 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019329    5005 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 01:51:13.027588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019333    5005 flags.go:64] FLAG: --max-pods="110"
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019336    5005 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019340    5005 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019343    5005 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019346    5005 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019349    5005 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019352    5005 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019355    5005 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019362    5005 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019368    5005 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019373    5005 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019378    5005 flags.go:64] FLAG: --pod-cidr=""
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019382    5005 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019387    5005 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019394    5005 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019398    5005 flags.go:64] FLAG: --port="10250"
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019403    5005 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019408    5005 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-03c7b893505f44a8c"
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019414    5005 flags.go:64] FLAG: --qos-reserved=""
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019419    5005 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019424    5005 flags.go:64] FLAG: --register-node="true"
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019429    5005 flags.go:64] FLAG: --register-with-taints=""
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019435    5005 flags.go:64] FLAG: --registry-burst="10"
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019440    5005 flags.go:64] FLAG: --registry-qps="5"
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019445    5005 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019451    5005 flags.go:64] FLAG: --reserved-memory=""
Nov 03 01:51:13.028299 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019457    5005 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019460    5005 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019464    5005 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019468    5005 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019472    5005 flags.go:64] FLAG: --runonce="false"
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019477    5005 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019482    5005 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019487    5005 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019492    5005 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019496    5005 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019501    5005 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019506    5005 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019510    5005 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019515    5005 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019520    5005 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019526    5005 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019531    5005 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019536    5005 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019540    5005 flags.go:64] FLAG: --system-cgroups=""
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019545    5005 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019556    5005 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019560    5005 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019565    5005 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019572    5005 flags.go:64] FLAG: --tls-min-version=""
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019577    5005 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 01:51:13.028919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019581    5005 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 01:51:13.029783 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019585    5005 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 01:51:13.029783 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019590    5005 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 01:51:13.029783 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019594    5005 flags.go:64] FLAG: --v="2"
Nov 03 01:51:13.029783 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019601    5005 flags.go:64] FLAG: --version="false"
Nov 03 01:51:13.029783 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019607    5005 flags.go:64] FLAG: --vmodule=""
Nov 03 01:51:13.029783 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019613    5005 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 01:51:13.029783 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.019618    5005 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 01:51:13.029783 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019806    5005 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:51:13.029783 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019812    5005 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:51:13.029783 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019817    5005 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:51:13.029783 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019822    5005 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:51:13.029783 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019826    5005 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:51:13.029783 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019829    5005 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:51:13.029783 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019833    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:51:13.029783 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019836    5005 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:51:13.029783 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019838    5005 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:51:13.029783 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019841    5005 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:51:13.029783 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019844    5005 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:51:13.029783 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019846    5005 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:51:13.029783 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019849    5005 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:51:13.029783 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019853    5005 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:51:13.030344 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019858    5005 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:51:13.030344 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019861    5005 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:51:13.030344 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019864    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:51:13.030344 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019867    5005 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:51:13.030344 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019869    5005 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:51:13.030344 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019875    5005 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:51:13.030344 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019878    5005 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:51:13.030344 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019881    5005 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:51:13.030344 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019884    5005 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:51:13.030344 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019886    5005 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:51:13.030344 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019889    5005 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:51:13.030344 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019891    5005 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:51:13.030344 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019894    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:51:13.030344 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019897    5005 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:51:13.030344 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019900    5005 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:51:13.030344 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019902    5005 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:51:13.030344 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019905    5005 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:51:13.030344 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019907    5005 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:51:13.030344 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019910    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:51:13.030344 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019913    5005 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:51:13.031022 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019916    5005 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:51:13.031022 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019923    5005 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:51:13.031022 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019927    5005 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:51:13.031022 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019929    5005 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:51:13.031022 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019932    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:51:13.031022 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019935    5005 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:51:13.031022 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019938    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:51:13.031022 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019941    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:51:13.031022 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019943    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:51:13.031022 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019946    5005 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:51:13.031022 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019949    5005 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:51:13.031022 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019952    5005 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:51:13.031022 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019956    5005 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:51:13.031022 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019958    5005 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:51:13.031022 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019961    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:51:13.031022 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019964    5005 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:51:13.031022 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019966    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:51:13.031022 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019971    5005 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:51:13.031022 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019974    5005 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:51:13.031482 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019977    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:51:13.031482 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019979    5005 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:51:13.031482 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019982    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:51:13.031482 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019985    5005 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:51:13.031482 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019988    5005 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:51:13.031482 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019990    5005 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:51:13.031482 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019993    5005 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:51:13.031482 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019996    5005 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:51:13.031482 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.019999    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:51:13.031482 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020001    5005 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:51:13.031482 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020004    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:51:13.031482 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020007    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:51:13.031482 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020009    5005 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:51:13.031482 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020012    5005 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:51:13.031482 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020014    5005 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:51:13.031482 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020017    5005 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:51:13.031482 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020020    5005 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:51:13.031482 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020023    5005 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:51:13.031482 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020025    5005 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:51:13.031482 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020028    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:51:13.032109 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020031    5005 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:51:13.032109 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020034    5005 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:51:13.032109 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020037    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:51:13.032109 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020039    5005 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:51:13.032109 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020042    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:51:13.032109 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020046    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:51:13.032109 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020048    5005 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:51:13.032109 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020051    5005 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:51:13.032109 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020053    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:51:13.032109 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020056    5005 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:51:13.032109 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020060    5005 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:51:13.032109 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020062    5005 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:51:13.032109 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020065    5005 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:51:13.032109 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020068    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:51:13.032109 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020071    5005 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:51:13.032109 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020074    5005 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:51:13.032109 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020076    5005 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:51:13.032109 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020079    5005 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:51:13.032109 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020082    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:51:13.032109 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020084    5005 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:51:13.032655 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020087    5005 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:51:13.032655 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020090    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:51:13.032655 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020092    5005 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:51:13.032655 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020108    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:51:13.032655 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020111    5005 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:51:13.032655 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020113    5005 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:51:13.032655 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020116    5005 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:51:13.032655 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020118    5005 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:51:13.032655 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020122    5005 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:51:13.032655 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020124    5005 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:51:13.032655 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020127    5005 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:51:13.032655 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020129    5005 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:51:13.032655 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020132    5005 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:51:13.032655 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.020158    5005 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 01:51:13.032655 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020198    5005 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:51:13.032655 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020203    5005 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:51:13.033146 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020206    5005 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:51:13.033146 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020210    5005 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:51:13.033146 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020213    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:51:13.033146 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020216    5005 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:51:13.033146 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020219    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:51:13.033146 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020222    5005 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:51:13.033146 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020225    5005 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:51:13.033146 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020228    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:51:13.033146 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020231    5005 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:51:13.033146 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020234    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:51:13.033146 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020236    5005 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:51:13.033146 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020239    5005 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:51:13.033146 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020242    5005 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:51:13.033146 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020244    5005 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:51:13.033146 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020247    5005 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:51:13.033146 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020250    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:51:13.033146 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020252    5005 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:51:13.033146 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020255    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:51:13.033146 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020258    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:51:13.033616 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020260    5005 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:51:13.033616 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020263    5005 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:51:13.033616 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020265    5005 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:51:13.033616 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020268    5005 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:51:13.033616 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020271    5005 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:51:13.033616 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020273    5005 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:51:13.033616 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020277    5005 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:51:13.033616 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020279    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:51:13.033616 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020282    5005 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:51:13.033616 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020284    5005 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:51:13.033616 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020287    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:51:13.033616 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020289    5005 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:51:13.033616 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020292    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:51:13.033616 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020294    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:51:13.033616 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020297    5005 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:51:13.033616 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020300    5005 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:51:13.033616 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020302    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:51:13.033616 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020305    5005 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:51:13.033616 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020307    5005 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:51:13.033616 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020310    5005 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:51:13.034312 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020313    5005 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:51:13.034312 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020316    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:51:13.034312 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020318    5005 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:51:13.034312 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020321    5005 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:51:13.034312 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020323    5005 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:51:13.034312 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020325    5005 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:51:13.034312 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020328    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:51:13.034312 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020331    5005 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:51:13.034312 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020333    5005 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:51:13.034312 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020336    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:51:13.034312 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020338    5005 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:51:13.034312 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020341    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:51:13.034312 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020343    5005 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:51:13.034312 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020346    5005 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:51:13.034312 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020348    5005 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:51:13.034312 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020351    5005 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:51:13.034312 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020353    5005 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:51:13.034312 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020356    5005 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:51:13.034312 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020358    5005 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:51:13.034836 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020361    5005 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:51:13.034836 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020364    5005 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:51:13.034836 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020366    5005 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:51:13.034836 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020368    5005 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:51:13.034836 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020373    5005 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:51:13.034836 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020377    5005 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:51:13.034836 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020380    5005 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:51:13.034836 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020383    5005 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:51:13.034836 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020386    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:51:13.034836 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020388    5005 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:51:13.034836 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020391    5005 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:51:13.034836 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020393    5005 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:51:13.034836 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020396    5005 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:51:13.034836 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020398    5005 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:51:13.034836 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020401    5005 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:51:13.034836 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020405    5005 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:51:13.034836 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020409    5005 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:51:13.034836 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020412    5005 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:51:13.034836 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020415    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:51:13.035317 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020417    5005 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:51:13.035317 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020420    5005 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:51:13.035317 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020423    5005 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:51:13.035317 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020425    5005 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:51:13.035317 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020428    5005 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:51:13.035317 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020430    5005 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:51:13.035317 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020433    5005 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:51:13.035317 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020436    5005 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:51:13.035317 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020438    5005 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:51:13.035317 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020441    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:51:13.035317 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020444    5005 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:51:13.035317 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020446    5005 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:51:13.035317 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020449    5005 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:51:13.035317 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020451    5005 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:51:13.035317 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020454    5005 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:51:13.035317 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020457    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:51:13.035317 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020460    5005 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:51:13.035317 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020462    5005 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:51:13.035317 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020465    5005 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:51:13.035317 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020468    5005 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:51:13.035920 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020470    5005 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:51:13.035920 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020473    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:51:13.035920 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020475    5005 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:51:13.035920 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020478    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:51:13.035920 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020481    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:51:13.035920 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020483    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:51:13.035920 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.020486    5005 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:51:13.035920 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.023992    5005 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 01:51:13.035920 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.024007    5005 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 01:51:13.035920 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.024019    5005 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 01:51:13.035920 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.024024    5005 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 01:51:13.035920 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024081    5005 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:51:13.035920 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024086    5005 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:51:13.035920 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024091    5005 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:51:13.035920 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024094    5005 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:51:13.035920 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024097    5005 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:51:13.035920 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024100    5005 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:51:13.035920 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024103    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:51:13.035920 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024106    5005 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:51:13.036380 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024109    5005 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:51:13.036380 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024112    5005 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:51:13.036380 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024115    5005 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:51:13.036380 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024117    5005 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:51:13.036380 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024120    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:51:13.036380 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024122    5005 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:51:13.036380 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024125    5005 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:51:13.036380 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024128    5005 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:51:13.036380 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024130    5005 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:51:13.036380 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024133    5005 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:51:13.036380 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024148    5005 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:51:13.036380 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024151    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:51:13.036380 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024154    5005 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:51:13.036380 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024158    5005 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:51:13.036380 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024162    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:51:13.036380 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024166    5005 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:51:13.036380 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024169    5005 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:51:13.036380 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024172    5005 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:51:13.036380 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024175    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:51:13.036892 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024178    5005 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:51:13.036892 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024181    5005 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:51:13.036892 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024184    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:51:13.036892 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024187    5005 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:51:13.036892 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024189    5005 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:51:13.036892 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024192    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:51:13.036892 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024194    5005 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:51:13.036892 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024197    5005 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:51:13.036892 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024199    5005 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:51:13.036892 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024202    5005 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:51:13.036892 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024204    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:51:13.036892 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024207    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:51:13.036892 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024210    5005 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:51:13.036892 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024212    5005 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:51:13.036892 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024214    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:51:13.036892 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024217    5005 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:51:13.036892 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024219    5005 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:51:13.036892 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024222    5005 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:51:13.036892 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024224    5005 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:51:13.036892 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024227    5005 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:51:13.037355 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024230    5005 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:51:13.037355 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024232    5005 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:51:13.037355 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024234    5005 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:51:13.037355 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024237    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:51:13.037355 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024240    5005 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:51:13.037355 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024242    5005 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:51:13.037355 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024245    5005 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:51:13.037355 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024247    5005 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:51:13.037355 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024251    5005 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:51:13.037355 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024255    5005 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:51:13.037355 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024258    5005 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:51:13.037355 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024261    5005 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:51:13.037355 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024264    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:51:13.037355 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024267    5005 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:51:13.037355 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024270    5005 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:51:13.037355 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024273    5005 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:51:13.037355 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024275    5005 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:51:13.037355 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024278    5005 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:51:13.037355 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024281    5005 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:51:13.037355 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024284    5005 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:51:13.037835 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024287    5005 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:51:13.037835 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024289    5005 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:51:13.037835 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024292    5005 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:51:13.037835 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024294    5005 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:51:13.037835 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024297    5005 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:51:13.037835 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024300    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:51:13.037835 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024302    5005 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:51:13.037835 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024305    5005 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:51:13.037835 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024307    5005 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:51:13.037835 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024310    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:51:13.037835 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024312    5005 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:51:13.037835 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024315    5005 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:51:13.037835 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024317    5005 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:51:13.037835 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024320    5005 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:51:13.037835 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024322    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:51:13.037835 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024325    5005 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:51:13.037835 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024328    5005 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:51:13.037835 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024331    5005 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:51:13.037835 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024333    5005 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:51:13.038284 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024336    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:51:13.038284 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024339    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:51:13.038284 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024341    5005 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:51:13.038284 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024344    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:51:13.038284 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024346    5005 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:51:13.038284 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024349    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:51:13.038284 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024352    5005 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:51:13.038284 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024355    5005 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:51:13.038284 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024357    5005 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:51:13.038284 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024360    5005 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:51:13.038284 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024363    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:51:13.038284 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024365    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:51:13.038284 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024368    5005 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:51:13.038284 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024370    5005 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:51:13.038284 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024373    5005 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:51:13.038284 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024375    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:51:13.038284 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024377    5005 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:51:13.038284 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024380    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:51:13.038284 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024382    5005 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:51:13.038284 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024385    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:51:13.038757 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.024411    5005 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 01:51:13.038757 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.024721    5005 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:51:13.038757 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025092    5005 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:51:13.038757 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025096    5005 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:51:13.038757 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025099    5005 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:51:13.038757 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025102    5005 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:51:13.038757 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025105    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:51:13.038757 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025108    5005 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:51:13.038757 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025111    5005 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:51:13.038757 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025114    5005 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:51:13.038757 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025117    5005 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:51:13.038757 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025119    5005 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:51:13.038757 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025123    5005 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:51:13.038757 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025133    5005 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:51:13.038757 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025138    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:51:13.038757 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025142    5005 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:51:13.039166 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025147    5005 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:51:13.039166 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025152    5005 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:51:13.039166 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025156    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:51:13.039166 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025162    5005 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:51:13.039166 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025167    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:51:13.039166 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025171    5005 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:51:13.039166 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025176    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:51:13.039166 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025181    5005 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:51:13.039166 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025186    5005 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:51:13.039166 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025195    5005 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:51:13.039166 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025199    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:51:13.039166 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025204    5005 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:51:13.039166 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025208    5005 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:51:13.039166 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025213    5005 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:51:13.039166 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025217    5005 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:51:13.039166 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025221    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:51:13.039166 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025225    5005 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:51:13.039166 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025230    5005 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:51:13.039166 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025234    5005 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:51:13.039166 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025238    5005 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:51:13.039636 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025242    5005 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:51:13.039636 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025246    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:51:13.039636 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025255    5005 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:51:13.039636 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025259    5005 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:51:13.039636 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025264    5005 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:51:13.039636 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025268    5005 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:51:13.039636 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025272    5005 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:51:13.039636 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025276    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:51:13.039636 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025280    5005 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:51:13.039636 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025285    5005 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:51:13.039636 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025289    5005 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:51:13.039636 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025293    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:51:13.039636 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025297    5005 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:51:13.039636 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025301    5005 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:51:13.039636 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025306    5005 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:51:13.039636 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025315    5005 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:51:13.039636 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025320    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:51:13.039636 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025325    5005 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:51:13.039636 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025329    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:51:13.039636 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025333    5005 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:51:13.040200 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025338    5005 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:51:13.040200 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025342    5005 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:51:13.040200 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025346    5005 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:51:13.040200 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025351    5005 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:51:13.040200 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025355    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:51:13.040200 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025360    5005 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:51:13.040200 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025364    5005 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:51:13.040200 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025368    5005 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:51:13.040200 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025378    5005 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:51:13.040200 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025382    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:51:13.040200 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025386    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:51:13.040200 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025391    5005 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:51:13.040200 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025395    5005 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:51:13.040200 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025400    5005 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:51:13.040200 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025404    5005 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:51:13.040200 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025408    5005 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:51:13.040200 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025413    5005 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:51:13.040200 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025420    5005 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:51:13.040200 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025426    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:51:13.040879 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025433    5005 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:51:13.040879 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025444    5005 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:51:13.040879 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025449    5005 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:51:13.040879 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025454    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:51:13.040879 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025458    5005 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:51:13.040879 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025462    5005 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:51:13.040879 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025467    5005 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:51:13.040879 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025471    5005 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:51:13.040879 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025475    5005 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:51:13.040879 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025479    5005 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:51:13.040879 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025484    5005 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:51:13.040879 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025489    5005 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:51:13.040879 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025493    5005 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:51:13.040879 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025501    5005 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:51:13.040879 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025505    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:51:13.040879 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025510    5005 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:51:13.040879 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025514    5005 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:51:13.040879 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025519    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:51:13.040879 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025523    5005 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:51:13.041630 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025527    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:51:13.041630 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025532    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:51:13.041630 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025536    5005 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:51:13.041630 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025540    5005 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:51:13.041630 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025545    5005 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:51:13.041630 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025548    5005 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:51:13.041630 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025553    5005 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:51:13.041630 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025562    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:51:13.041630 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025566    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:51:13.041630 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025570    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:51:13.041630 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025575    5005 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:51:13.041630 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025579    5005 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:51:13.041630 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025584    5005 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:51:13.041630 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.025678    5005 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 01:51:13.041630 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025909    5005 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:51:13.041630 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025918    5005 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:51:13.042313 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025922    5005 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:51:13.042313 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025925    5005 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:51:13.042313 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025930    5005 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:51:13.042313 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025933    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:51:13.042313 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025935    5005 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:51:13.042313 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025938    5005 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:51:13.042313 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025941    5005 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:51:13.042313 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025943    5005 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:51:13.042313 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025946    5005 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:51:13.042313 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025949    5005 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:51:13.042313 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025952    5005 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:51:13.042313 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025954    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:51:13.042313 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025957    5005 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:51:13.042313 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025960    5005 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:51:13.042313 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025963    5005 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:51:13.042313 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025966    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:51:13.042313 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025968    5005 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:51:13.042313 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025971    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:51:13.042313 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025973    5005 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:51:13.042313 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025976    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:51:13.043441 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025979    5005 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:51:13.043441 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025981    5005 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:51:13.043441 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025984    5005 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:51:13.043441 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025987    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:51:13.043441 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025989    5005 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:51:13.043441 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025993    5005 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:51:13.043441 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025996    5005 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:51:13.043441 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.025999    5005 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:51:13.043441 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026001    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:51:13.043441 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026004    5005 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:51:13.043441 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026006    5005 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:51:13.043441 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026009    5005 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:51:13.043441 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026011    5005 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:51:13.043441 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026014    5005 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:51:13.043441 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026016    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:51:13.043441 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026019    5005 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:51:13.043441 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026021    5005 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:51:13.043441 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026024    5005 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:51:13.043441 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026026    5005 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:51:13.044287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026029    5005 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:51:13.044287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026031    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:51:13.044287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026034    5005 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:51:13.044287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026037    5005 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:51:13.044287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026040    5005 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:51:13.044287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026042    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:51:13.044287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026045    5005 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:51:13.044287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026048    5005 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:51:13.044287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026050    5005 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:51:13.044287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026055    5005 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:51:13.044287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026057    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:51:13.044287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026060    5005 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:51:13.044287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026062    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:51:13.044287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026065    5005 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:51:13.044287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026067    5005 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:51:13.044287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026070    5005 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:51:13.044287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026073    5005 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:51:13.044287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026075    5005 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:51:13.044287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026077    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:51:13.044287 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026080    5005 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:51:13.045130 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026084    5005 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:51:13.045130 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026086    5005 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:51:13.045130 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026089    5005 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:51:13.045130 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026091    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:51:13.045130 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026094    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:51:13.045130 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026097    5005 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:51:13.045130 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026099    5005 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:51:13.045130 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026102    5005 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:51:13.045130 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026104    5005 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:51:13.045130 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026107    5005 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:51:13.045130 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026109    5005 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:51:13.045130 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026113    5005 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:51:13.045130 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026118    5005 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:51:13.045130 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026122    5005 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:51:13.045130 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026125    5005 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:51:13.045130 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026128    5005 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:51:13.045130 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026134    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:51:13.045130 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026137    5005 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:51:13.045130 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026140    5005 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:51:13.045991 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026142    5005 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:51:13.045991 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026145    5005 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:51:13.045991 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026147    5005 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:51:13.045991 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026150    5005 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:51:13.045991 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026152    5005 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:51:13.045991 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026155    5005 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:51:13.045991 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026158    5005 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:51:13.045991 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026160    5005 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:51:13.045991 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026163    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:51:13.045991 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026166    5005 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:51:13.045991 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026168    5005 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:51:13.045991 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026171    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:51:13.045991 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026175    5005 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:51:13.045991 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026178    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:51:13.045991 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026181    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:51:13.045991 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026184    5005 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:51:13.045991 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026186    5005 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:51:13.045991 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026189    5005 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:51:13.045991 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026191    5005 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:51:13.045991 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026194    5005 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:51:13.046887 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026196    5005 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:51:13.046887 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026199    5005 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:51:13.046887 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026201    5005 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:51:13.046887 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026204    5005 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:51:13.046887 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026207    5005 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:51:13.046887 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:13.026209    5005 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:51:13.046887 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.026277    5005 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 01:51:13.046887 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.027232    5005 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 01:51:13.046887 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.027280    5005 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 01:51:13.046887 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.027458    5005 server.go:1014] "Starting client certificate rotation"
Nov 03 01:51:13.046887 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.027604    5005 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 01:51:13.046887 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.027664    5005 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 01:45:30 +0000 UTC" deadline="2028-04-05 01:30:05.787701587 +0000 UTC"
Nov 03 01:51:13.046887 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.027728    5005 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12455h38m52.759977353s"
Nov 03 01:51:13.046887 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.027938    5005 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 01:51:13.046887 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.028982    5005 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 01:51:13.046887 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.030177    5005 log.go:25] "Validated CRI v1 runtime API"
Nov 03 01:51:13.047560 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.031796    5005 log.go:25] "Validated CRI v1 image API"
Nov 03 01:51:13.047560 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.032510    5005 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 01:51:13.047560 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.034825    5005 fs.go:135] Filesystem UUIDs: map[557a8415-5443-4dfd-8bac-4442f77ff790:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e4ed0fdc-bd28-4343-a9fa-a720d63f539b:/dev/nvme0n1p3]
Nov 03 01:51:13.047715 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.034848    5005 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c9e32359e3ef256b0e8b244ad8f92a56f410da75819b00fd547aa422d729cb4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c9e32359e3ef256b0e8b244ad8f92a56f410da75819b00fd547aa422d729cb4/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16c546e51f74fc98b52aef4f1f105191f19b7af9ee68374cd6c570e88ee5b764/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16c546e51f74fc98b52aef4f1f105191f19b7af9ee68374cd6c570e88ee5b764/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b1cabad9b401aeae3bd442116212c87000eae80b7baad5c9ac7d4cf503832f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b1cabad9b401aeae3bd442116212c87000eae80b7baad5c9ac7d4cf503832f3/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d56f450ef97694ee4750e860271e17224a4d8210b40a24273628c3907ad8860/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d56f450ef97694ee4750e860271e17224a4d8210b40a24273628c3907ad8860/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b60f291a818eea4d3505e73812647b86256e017b112bfafc09a0e703be5fe16/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b60f291a818eea4d3505e73812647b86256e017b112bfafc09a0e703be5fe16/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c243069849fdbd584653523b58a37178540579d86c55ed2731bcfe14e30bffe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c243069849fdbd584653523b58a37178540579d86c55ed2731bcfe14e30bffe/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/47b4e6f25f392ebd934f9fa89d07ddb5b2ffa976c3422bc60f8c4ddaca5397ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/47b4e6f25f392ebd934f9fa89d07ddb5b2ffa976c3422bc60f8c4ddaca5397ad/userdata/shm major:0 minor:281 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ecfcc08a256f4626dbbeccb2c8ed8bb53b8fde82e97487eb2f786119c8230c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ecfcc08a256f4626dbbeccb2c8ed8bb53b8fde82e97487eb2f786119c8230c2/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/68c010f0f76f47a515f05930ada535e464568c6ac669fdd14e110b17f7a77c3a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/68c010f0f76f47a515f05930ada535e464568c6ac669fdd14e110b17f7a77c3a/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c6141e7f39691eba8f6633218ff32e37cd22cc731def6a22005e4618c1fb3be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c6141e7f39691eba8f6633218ff32e37cd22cc731def6a22005e4618c1fb3be/userdata/shm major:0 minor:343 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d0ceec34113f5a8a9eb64e5c5e2899b435a56fa1464f3bb1c6292ddbff3a336/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d0ceec34113f5a8a9eb64e5c5e2899b435a56fa1464f3bb1c6292ddbff3a336/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7c4527b6974ef6c6920a016d149e89500669f178085b5e49dbfd029f2617c0fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7c4527b6974ef6c6920a016d149e89500669f178085b5e49dbfd029f2617c0fa/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80c2f3b680f02615b275e52c25d8dca0b759e27c479f7ef19bc486e6d75cd70b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80c2f3b680f02615b275e52c25d8dca0b759e27c479f7ef19bc486e6d75cd70b/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a2b0a911b8a92222130cb200c593379e9a7975d1527e41c6b12a12dae6af66e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a2b0a911b8a92222130cb200c593379e9a7975d1527e41c6b12a12dae6af66e/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99c457ae3480cd320c269741628c5969c011a89322c106e507d3670364521f08/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99c457ae3480cd320c269741628c5969c011a89322c106e507d3670364521f08/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d6e9127f4b7588b09b87bdcb7399d252e4fa32a62cfd57756e7a7246766a2d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d6e9127f4b7588b09b87bdcb7399d252e4fa32a62cfd57756e7a7246766a2d3/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a79a11b92c5252da0a40c17154db0cf7ca937cde26f1e2192e5143f2bd2c2b6c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a79a11b92c5252da0a40c17154db0cf7ca937cde26f1e2192e5143f2bd2c2b6c/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b02ad7dea43ede949cc8a9433f233c2a12cb95d0768b423eb90ce3478e7c1b85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b02ad7dea43ede949cc8a9433f233c2a12cb95d0768b423eb90ce3478e7c1b85/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba70ce368c30cd7c278ff369764b964bbb9d236b809ad498811a47824475f279/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba70ce368c30cd7c278ff369764b964bbb9d236b809ad498811a47824475f279/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1d497aa54f9d2bfadefb60b3ecd9f69712b6dcc0d918a57a7de9b2170079aa3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1d497aa54f9d2bfadefb60b3ecd9f69712b6dcc0d918a57a7de9b2170079aa3/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1f69404f3c50f789279e5a3757d9a8b1b4787b5e39d06df736c090657cfed25/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1f69404f3c50f789279e5a3757d9a8b1b4787b5e39d06df736c090657cfed25/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5ccfb7e0ba4c33f7f2ac59a18907866c0bc8256d609daf39884235e007e6176/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5ccfb7e0ba4c33f7f2ac59a18907866c0bc8256d609daf39884235e007e6176/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e59b0b9e967622046c45b1bbd1249b6ea96e8186fc6340e94dd34ee27174e7ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e59b0b9e967622046c45b1bbd1249b6ea96e8186fc6340e94dd34ee27174e7ac/userdata/shm major:0 minor:332 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ebe7cc246a6948855e1b3fa82fe1be3cdae3ef69f6aa1d6ac06f80a116c5984f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ebe7cc246a6948855e1b3fa82fe1be3cdae3ef69f6aa1d6ac06f80a116c5984f/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee5c4e58b1d36c1d73804b6b742b1354d358c2693d05893eae4ee53d65b508c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee5c4e58b1d36c1d73804b6b742b1354d358c2693d05893eae4ee53d65b508c3/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fcd57864758c6e788a88dea6c88fb474427145eb04c28fd86dfce47b14f6d3b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fcd57864758c6e788a88dea6c88fb474427145eb04c28fd86dfce47b14f6d3b0/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431/volumes/kubernetes.io~projected/kube-api-access-f2d5x:{mountpoint:/var/lib/kubelet/pods/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431/volumes/kubernetes.io~projected/kube-api-access-f2d5x major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/108d9224-ccd3-4925-a65b-0d59e938ff72/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/108d9224-ccd3-4925-a65b-0d59e938ff72/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/108d9224-ccd3-4925-a65b-0d59e938ff72/volumes/kubernetes.io~projected/kube-api-access-cg5wd:{mountpoint:/var/lib/kubelet/pods/108d9224-ccd3-4925-a65b-0d59e938ff72/volumes/kubernetes.io~projected/kube-api-access-cg5wd major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/108d9224-ccd3-4925-a65b-0d59e938ff72/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/108d9224-ccd3-4925-a65b-0d59e938ff72/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/108d9224-ccd3-4925-a65b-0d59e938ff72/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/108d9224-ccd3-4925-a65b-0d59e938ff72/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15face97-689e-4fef-86c0-17a2a85829db/volumes/kubernetes.io~projected/kube-api-access-zqsmb:{mountpoint:/var/lib/kubelet/pods/15face97-689e-4fef-86c0-17a2a85829db/volumes/kubernetes.io~projected/kube-api-access-zqsmb major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15face97-689e-4fef-86c0-17a2a85829db/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/15face97-689e-4fef-86c0-17a2a85829db/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8/volumes/kubernetes.io~projected/kube-api-access-5h48h:{mountpoint:/var/lib/kubelet/pods/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8/volumes/kubernetes.io~projected/kube-api-access-5h48h major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f0923bb-db25-4266-ad8d-2e1cfb7f602b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/1f0923bb-db25-4266-ad8d-2e1cfb7f602b/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2555270f-a2ea-4d11-8dff-d130f9786e3d/volumes/kubernetes.io~projected/kube-api-access-q5lg2:{mountpoint:/var/lib/kubelet/pods/2555270f-a2ea-4d11-8dff-d130f9786e3d/volumes/kubernetes.io~projected/kube-api-access-q5lg2 major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/354d7018-887a-497f-b74f-195c8c88e184/volumes/kubernetes.io~projected/kube-api-access-fg6dt:{mountpoint:/var/lib/kubelet/pods/354d7018-887a-497f-b74f-195c8c88e184/volumes/kubernetes.io~projected/kube-api-access-fg6dt major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/354d7018-887a-497f-b74f-195c8c88e184/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/354d7018-887a-497f-b74f-195c8c88e184/volumes/kubernetes.io~secret/serving-cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38de413a-873a-4288-85f5-c453581a53d6/volumes/kubernetes.io~projected/kube-api-access-9ln9h:{mountpoint:/var/lib/kubelet/pods/38de413a-873a-4288-85f5-c453581a53d6/volumes/kubernetes.io~projected/kube-api-access-9ln9h major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a6655a6-9079-4927-8e6c-09d075de438c/volumes/kubernetes.io~projected/kube-api-access-btsvv:{mountpoint:/var/lib/kubelet/pods/3a6655a6-9079-4927-8e6c-09d075de438c/volumes/kubernetes.io~projected/kube-api-access-btsvv major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c/volumes/kubernetes.io~projected/kube-api-access-hq5lq:{mountpoint:/var/lib/kubelet/pods/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c/volumes/kubernetes.io~projected/kube-api-access-hq5lq major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48604fe3-9728-461e-ae37-8208b424e149/volumes/kubernetes.io~projected/kube-api-access-zxvzg:{mountpoint:/var/lib/kubelet/pods/48604fe3-9728-461e-ae37-8208b424e149/volumes/kubernetes.io~projected/kube-api-access-zxvzg major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54ba28bb-4bab-4c0d-aa00-8b3bce675f23/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/54ba28bb-4bab-4c0d-aa00-8b3bce675f23/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54ba28bb-4bab-4c0d-aa00-8b3bce675f23/volumes/kubernetes.io~projected/kube-api-access-flcc2:{mountpoint:/var/lib/kubelet/pods/54ba28bb-4bab-4c0d-aa00-8b3bce675f23/volumes/kubernetes.io~projected/kube-api-access-flcc2 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54ba28bb-4bab-4c0d-aa00-8b3bce675f23/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/54ba28bb-4bab-4c0d-aa00-8b3bce675f23/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62150808-f089-4723-aac5-d17d3a803abe/volumes/kubernetes.io~projected/kube-api-access-g5jp6:{mountpoint:/var/lib/kubelet/pods/62150808-f089-4723-aac5-d17d3a803abe/volumes/kubernetes.io~projected/kube-api-access-g5jp6 major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62c3ca06-b2a4-49d7-84db-1390277be2c8/volumes/kubernetes.io~projected/kube-api-access-z46t7:{mountpoint:/var/lib/kubelet/pods/62c3ca06-b2a4-49d7-84db-1390277be2c8/volumes/kubernetes.io~projected/kube-api-access-z46t7 major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/665f9d02-067e-4506-9793-cc7f17eb95bb/volumes/kubernetes.io~projected/kube-api-access-cvkdd:{mountpoint:/var/lib/kubelet/pods/665f9d02-067e-4506-9793-cc7f17eb95bb/volumes/kubernetes.io~projected/kube-api-access-cvkdd major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79fe02fa-99ce-4222-b569-6b323df47ab4/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/79fe02fa-99ce-4222-b569-6b323df47ab4/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a9e57fd-a6e6-4ddb-aa08-ac1390403544/volumes/kubernetes.io~projected/kube-api-access-rzfk9:{mountpoint:/var/lib/kubelet/pods/7a9e57fd-a6e6-4ddb-aa08-ac1390403544/volumes/kubernetes.io~projected/kube-api-access-rzfk9 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a9e57fd-a6e6-4ddb-aa08-ac1390403544/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/7a9e57fd-a6e6-4ddb-aa08-ac1390403544/volumes/kubernetes.io~secret/default-certificate major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a9e57fd-a6e6-4ddb-aa08-ac1390403544/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/7a9e57fd-a6e6-4ddb-aa08-ac1390403544/volumes/kubernetes.io~secret/stats-auth major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8684a429-234f-4537-917e-5b49509bbdb5/volumes/kubernetes.io~projected/kube-api-access-h6lhl:{mountpoint:/var/lib/kubelet/pods/8684a429-234f-4537-917e-5b49509bbdb5/volumes/kubernetes.io~projected/kube-api-access-h6lhl major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9260d15b-9772-4cb4-8588-8e82f89f39b8/volumes/kubernetes.io~projected/kube-api-access-hhg5z:{mountpoint:/var/lib/kubelet/pods/9260d15b-9772-4cb4-8588-8e82f89f39b8/volumes/kubernetes.io~projected/kube-api-access-hhg5z major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c546cea-b474-44cf-84a5-b89784bda1d6/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/9c546cea-b474-44cf-84a5-b89784bda1d6/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c546cea-b474-44cf-84a5-b89784bda1d6/volumes/kubernetes.io~projected/kube-api-access-85997:{mountpoint:/var/lib/kubelet/pods/9c546cea-b474-44cf-84a5-b89784bda1d6/volumes/kubernetes.io~projected/kube-api-access-85997 major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c546cea-b474-44cf-84a5-b89784bda1d6/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/9c546cea-b474-44cf-84a5-b89784bda1d6/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c546cea-b474-44cf-84a5-b89784bda1d6/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/9c546cea-b474-44cf-84a5-b89784bda1d6/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/volumes/kubernetes.io~projected/kube-api-access-6t8q9:{mountpoint:/var/lib/kubelet/pods/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/volumes/kubernetes.io~projected/kube-api-access-6t8q9 major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/volumes/kubernetes.io~secret/serving-cert major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b25bd7be-290a-499a-a81d-c005793eace2/volumes/kubernetes.io~projected/kube-api-access-8c6wq:{mountpoint:/var/lib/kubelet/pods/b25bd7be-290a-499a-a81d-c005793eace2/volumes/kubernetes.io~projected/kube-api-access-8c6wq major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b25bd7be-290a-499a-a81d-c005793eace2/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/b25bd7be-290a-499a-a81d-c005793eace2/volumes/kubernetes.io~secret/cert major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b57cc991-d758-435f-9238-0dac97f10a93/volumes/kubernetes.io~projected/kube-api-access-sl96f:{mountpoint:/var/lib/kubelet/pods/b57cc991-d758-435f-9238-0dac97f10a93/volumes/kubernetes.io~projected/kube-api-access-sl96f major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/baf5281f-5938-4417-b22f-2644034da7ef/volumes/kubernetes.io~projected/kube-api-access-mc9gg:{mountpoint:/var/lib/kubelet/pods/baf5281f-5938-4417-b22f-2644034da7ef/volumes/kubernetes.io~projected/kube-api-access-mc9gg major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/baf5281f-5938-4417-b22f-2644034da7ef/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/baf5281f-5938-4417-b22f-2644034da7ef/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be449663-2797-487f-8559-0a96c563ae3a/volumes/kubernetes.io~projected/kube-api-access-5z56p:{mountpoint:/var/lib/kubelet/pods/be449663-2797-487f-8559-0a96c563ae3a/volumes/kubernetes.io~projected/kube-api-access-5z56p major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7a013c2-18a4-4372-adcb-8dfafa673f9c/volumes/kubernetes.io~projected/kube-api-access-ltkx2:{mountpoint:/var/lib/kubelet/pods/c7a013c2-18a4-4372-adcb-8dfafa673f9c/volumes/kubernetes.io~projected/kube-api-access-ltkx2 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb8ca898-6f5f-4727-9d6d-d6aed65adfae/volumes/kubernetes.io~projected/kube-api-access-f76x5:{mountpoint:/var/lib/kubelet/pods/cb8ca898-6f5f-4727-9d6d-d6aed65adfae/volumes/kubernetes.io~projected/kube-api-access-f76x5 major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0de4046-bfed-4695-af0e-1ba806e6b031/volumes/kubernetes.io~projected/kube-api-access-dkrns:{mountpoint:/var/lib/kubelet/pods/d0de4046-bfed-4695-af0e-1ba806e6b031/volumes/kubernetes.io~projected/kube-api-access-dkrns major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0de4046-bfed-4695-af0e-1ba806e6b031/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/d0de4046-bfed-4695-af0e-1ba806e6b031/volumes/kubernetes.io~secret/signing-key major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e11ea02d-ac52-4318-9fdd-95c0bb171aed/volumes/kubernetes.io~projected/kube-api-access-2xbq7:{mountpoint:/var/lib/kubelet/pods/e11ea02d-ac52-4318-9fdd-95c0bb171aed/volumes/kubernetes.io~projected/kube-api-access-2xbq7 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e49068d1-9ae4-47b3-9a07-3d695a0a2402/volumes/kubernetes.io~projected/kube-api-access-jcpqh:{mountpoint:/var/lib/kubelet/pods/e49068d1-9ae4-47b3-9a07-3d695a0a2402/volumes/kubernetes.io~projected/kube-api-access-jcpqh major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef90f36a-279e-4bfa-a97d-70132a266670/volumes/kubernetes.io~projected/kube-api-access-kkw2h:{mountpoint:/var/lib/kubelet/pods/ef90f36a-279e-4bfa-a97d-70132a266670/volumes/kubernetes.io~projected/kube-api-access-kkw2h major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f30f499a-7d25-4ea7-aa1e-357ccbf0485f/volumes/kubernetes.io~projected/kube-api-access-qqqkf:{mountpoint:/var/lib/kubelet/pods/f30f499a-7d25-4ea7-aa1e-357ccbf0485f/volumes/kubernetes.io~projected/kube-api-access-qqqkf major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4425c17-5cc6-4afc-922c-d74694805196/volumes/kubernetes.io~projected/kube-api-access-8n62b:{mountpoint:/var/lib/kubelet/pods/f4425c17-5cc6-4afc-922c-d74694805196/volumes/kubernetes.io~projected/kube-api-access-8n62b major:0 minor:230 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/99ae11a4477f1c71a280b454b1021ff42fddb5141435bc7d57140af3317eb58f/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/b3ba83008645940ed99008d01abdec5994973efa97ae0447ef29d699395a1469/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/45095c6b657d8edc95e10f00f8ba0825e61236f223afd3f256eae883c6820136/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/c4c96cbe75f22c26d5c8f5504c5cf4ce3b0b64c1af293009e6137f1f6c0fbb7e/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/efeb54c54cecfbcd6f0cfa366784e369017cb85e3e952fef9c9e17c50a828f42/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/bce98b7454d4207d1c4ff16a4e723e1bd335ec7c45b7ded53a8230e495d8e920/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/9e3e31994c222dffadbe45fd94b66bac74dc967a3ba28a8155120674ebf3bf1b/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/08f787d0e2c3645ecc7ee25422d2b81997d26c2a409af95c5fdb8f8753b30647/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/849b5d59f47f8a32fc11995f35e1947e742198ad67e0afd4efb837edbe3bc531/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/73d4c38d195bb35e4dfcb2e3eff7d318ea7639f278c929fad741002d4900903f/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-135:{mountpoint:/var/lib/containers/storage/overlay/172415bb41952c34514390097070adac9b2bd01736b575256d6e7d8e3059d756/merged major:0 minor:135 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/987b6cb29841674b47f86689e925def9fe8ca378dbe795cc203f16aeb67f757b/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/6bd220698c56645809f0841eb9eebdcf819ea37029f84ec4767c8037b62da11c/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/39a50c5a6d26eb52003f6754ab1603f8a5e4cbf8eef1485c0b954bcd10e4a3e7/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/5023008ce8f1bc246efdb7000fc5b25cadd2f8bc18c083162f3d3e6191996740/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/c055ec0ef09696073b17662981e88349fbb794dfcd67b4c13010379f1b216c6d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/2ade6d738f2b6d1fd5723e41b923c9b448aee56a9e29421e1a4a26aa58ce6470/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/79101ab2cf07381e16204d3eadbcaae863bdc4ff57d01910f008baa163163859/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ff1dc138b71804af3abf33ff3d02e13367b9bfe04aa86d73971b2073cc27306e/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/678be9013059110dccb1139d32ac19169eed6a78355c36c05529f3767e282f73/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e51977a7f275306fe7388aa33d985f065677c265452837688309128c20e32ef3/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/03fd3d307bda2b86a90fb8d1eb6e892fc8cd4e59c15994907eb0c39b5d7bb7d8/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/39cf966e95c44756d5ad02d786a2291917387564c7b3ceb964d5be56453dc28b/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/7196f8cf9cc2a654c738293759c2e9e6e9dc5835e3c83e016314bc71a95274ca/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/0c82737fe7a9fed3e14c1a88e4ebea70c7a7749ee046aaecca2fb243497c6b7a/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/9d5668f4a10f0bbd7a8f84aa82d37fa1d0c14ba88f7176a459ff7594df75a14d/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/1fc2e5a7d380c7e747667616416e1c762162965e366e7bcb5316f3d1df1d664e/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/a0e98fa4520263a8ef7e0c1248a1f852ecb091b1d2c8067bf355dc8fa470bfdd/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/535e83aa90c0d2acd96fdb81c503a41e66872fc77125d0d987a875bab7a3fa7c/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/b45d9a4db5bc0f80425ee04f3e74446a26066bc370e6be14acd2f1f15d328b11/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/7c70edd90574f051253a7e45e9242ce2d860d266ecb2b40472046638453614c2/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/7b1283c24d166f79e27b2d12b4bab7662cffc2d2400e1f880390b2ca83765549/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/f7789b79fb60a7649fd687a155383c0f8f88e81d4797ff63b7df6c08638bfc12/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/7a462f46bb05e4b44d8e453693eb12cca59638bf4278c0e74bdbdccf6a7c5005/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/ccb8ea561dd324945355749508c41b021766fd30431ccebc2ed5d76321060283/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/11149c179aaa0955b5e33ba60bd27fb0e05a154bf0a9f00783b58153af89d1e7/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/f1251a7609224e93849737de2220925038fecab2aeb91d7fbc670a48515e4d44/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/d86f0c3fd2e54ceed17795bf5c08ddd955a26efd30bf4874f06d3847ac3c575d/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/933e818c1784c21f924c2c3f377bed5db6bd1707becd4f445c6bb821b237ee1e/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/c1fa76d4414660fbac84c3db366de42be5fb97c3d8bfac66ccb612ce08ab3250/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/22fc21d13b5f456a9a6fb7546bbfc8240643d2698e0fc57ce33b494ebcd31c77/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/588d06cce478c2c6427aec2bfa5f6daea0df24acffd0735c1d98ccdd1aef9891/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/e8bfdd03937e68573c9fdf23e0d6187e605fd59aef7585693480e52cac423579/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/00888aa915aed61550bf20a4689d94d72a2f268ac3f96f9c3ab36b84054c6693/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/05c92497fea5f4f7fe275c1cd6e4299d20918af68699981f448f136e2365d781/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/b504edfcb3db44e0a3509a1ea3ad010a46647172b9f88ffa63c16db9a3760708/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/4e8908817ea2f065f62ccb8168261123765abb22c8a4209774012cdebfff69c0/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/fb1b6b8eaaf61cd18025fc70aec553d5926659ac6d6d95923a20714751b7cd58/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/1e2df102d66cf4dac52556c9817791b3f3b32dae423738ba09a4f9d6b77e9c4b/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/e32b9d4cfb516cfe0f25157fd51b00048684209b93d70f3e8d142c52d0c0bbd1/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/da185cda896116d68c2770ac3a8800e43c8fcfbb0e0273bea42f22a1fe6b4cbe/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/c09ff9ffd3d1b6fd314eabbe95d91cd8825bcb2f7d37b7f43f68bce125abc126/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/29402de3d9c186cf017d42b8be0ba2411e3cc8bbb916edd9bb806f24e2a681f1/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/45a1f35d891b707974105dc8df2d79f81041d460db5ab7177e4779a0b45a9307/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/509b8f143941310b76d915a5876f5a16a1a3d6205225d5192521c4c20c0a67da/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/2fd8cb8867a3ae95428476cbc0c4d7f815b3d3f3e3ff62e57a19f5e48e008cfd/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/97e6f627ae7fbec4cc3ff6c34e41ba1370fdaf7980ee5548dee024e39846b136/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/e7d5df43674c4c059a7a23ef31507688a7495ba597224117f1bcc48c70154043/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/da7537499b69cac12a0bea4cfa5060dfcebc5d72e04933506346474154694f10/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/cde3f9e67a81ca8486cbc2099191e20dcb045f92c70f6d8928bdc2b7ae72375c/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 01:51:13.048108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.044432    5005 manager.go:217] Machine: {Timestamp:2026-11-03 01:51:13.043592352 +0000 UTC m=+0.057833709 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199870 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d7708420d8838b64bd98fb15f3304 SystemUUID:ec2d7708-420d-8838-b64b-d98fb15f3304 BootID:2bfd3b4a-86b6-4a98-bf0f-108cf1122432 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/baf5281f-5938-4417-b22f-2644034da7ef/volumes/kubernetes.io~projected/kube-api-access-mc9gg DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e59b0b9e967622046c45b1bbd1249b6ea96e8186fc6340e94dd34ee27174e7ac/userdata/shm DeviceMajor:0 DeviceMinor:332 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1f0923bb-db25-4266-ad8d-2e1cfb7f602b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a2b0a911b8a92222130cb200c593379e9a7975d1527e41c6b12a12dae6af66e/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c243069849fdbd584653523b58a37178540579d86c55ed2731bcfe14e30bffe/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d56f450ef97694ee4750e860271e17224a4d8210b40a24273628c3907ad8860/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/be449663-2797-487f-8559-0a96c563ae3a/volumes/kubernetes.io~projected/kube-api-access-5z56p DeviceMajor:0 DeviceMinor:318 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8684a429-234f-4537-917e-5b49509bbdb5/volumes/kubernetes.io~projected/kube-api-access-h6lhl DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/15face97-689e-4fef-86c0-17a2a85829db/volumes/kubernetes.io~projected/kube-api-access-zqsmb DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/108d9224-ccd3-4925-a65b-0d59e938ff72/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c9e32359e3ef256b0e8b244ad8f92a56f410da75819b00fd547aa422d729cb4/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b1cabad9b401aeae3bd442116212c87000eae80b7baad5c9ac7d4cf503832f3/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e49068d1-9ae4-47b3-9a07-3d695a0a2402/volumes/kubernetes.io~projected/kube-api-access-jcpqh DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ebe7cc246a6948855e1b3fa82fe1be3cdae3ef69f6aa1d6ac06f80a116c5984f/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7a9e57fd-a6e6-4ddb-aa08-ac1390403544/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/volumes/kubernetes.io~projected/kube-api-access-6t8q9 DeviceMajor:0 DeviceMinor:236 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c6141e7f39691eba8f6633218ff32e37cd22cc731def6a22005e4618c1fb3be/userdata/shm DeviceMajor:0 DeviceMinor:343 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8/volumes/kubernetes.io~projected/kube-api-access-5h48h DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431/volumes/kubernetes.io~projected/kube-api-access-f2d5x DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/15face97-689e-4fef-86c0-17a2a85829db/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7a9e57fd-a6e6-4ddb-aa08-ac1390403544/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/354d7018-887a-497f-b74f-195c8c88e184/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9c546cea-b474-44cf-84a5-b89784bda1d6/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9c546cea-b474-44cf-84a5-b89784bda1d6/volumes/kubernetes.io~projected/kube-api-access-85997 DeviceMajor:0 DeviceMinor:233 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5ccfb7e0ba4c33f7f2ac59a18907866c0bc8256d609daf39884235e007e6176/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1d497aa54f9d2bfadefb60b3ecd9f69712b6dcc0d918a57a7de9b2170079aa3/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62150808-f089-4723-aac5-d17d3a803abe/volumes/kubernetes.io~projected/kube-api-access-g5jp6 DeviceMajor:0 DeviceMinor:323 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7c4527b6974ef6c6920a016d149e89500669f178085b5e49dbfd029f2617c0fa/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b60f291a818eea4d3505e73812647b86256e017b112bfafc09a0e703be5fe16/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b02ad7dea43ede949cc8a9433f233c2a12cb95d0768b423eb90ce3478e7c1b85/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-135 DeviceMajor:0 DeviceMinor:135 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/665f9d02-067e-4506-9793-cc7f17eb95bb/volumes/kubernetes.io~projected/kube-api-access-cvkdd DeviceMajor:0 DeviceMinor:231 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16c546e51f74fc98b52aef4f1f105191f19b7af9ee68374cd6c570e88ee5b764/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba70ce368c30cd7c278ff369764b964bbb9d236b809ad498811a47824475f279/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99c457ae3480cd320c269741628c5969c011a89322c106e507d3670364521f08/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cb8ca898-6f5f-4727-9d6d-d6aed65adfae/volumes/kubernetes.io~projected/kube-api-access-f76x5 DeviceMajor:0 DeviceMinor:331 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/38de413a-873a-4288-85f5-c453581a53d6/volumes/kubernetes.io~projected/kube-api-access-9ln9h DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54ba28bb-4bab-4c0d-aa00-8b3bce675f23/volumes/kubernetes.io~projected/kube-api-access-flcc2 DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/baf5281f-5938-4417-b22f-2644034da7ef/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62c3ca06-b2a4-49d7-84db-1390277be2c8/volumes/kubernetes.io~projected/kube-api-access-z46t7 DeviceMajor:0 DeviceMinor:330 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/54ba28bb-4bab-4c0d-aa00-8b3bce675f23/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/54ba28bb-4bab-4c0d-aa00-8b3bce675f23/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee5c4e58b1d36c1d73804b6b742b1354d358c2693d05893eae4ee53d65b508c3/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9c546cea-b474-44cf-84a5-b89784bda1d6/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79fe02fa-99ce-4222-b569-6b323df47ab4/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:274 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e11ea02d-ac52-4318-9fdd-95c0bb171aed/volumes/kubernetes.io~projected/kube-api-access-2xbq7 DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ef90f36a-279e-4bfa-a97d-70132a266670/volumes/kubernetes.io~projected/kube-api-access-kkw2h DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b25bd7be-290a-499a-a81d-c005793eace2/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:237 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7a013c2-18a4-4372-adcb-8dfafa673f9c/volumes/kubernetes.io~projected/kube-api-access-ltkx2 DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7a9e57fd-a6e6-4ddb-aa08-ac1390403544/volumes/kubernetes.io~projected/kube-api-access-rzfk9 DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/47b4e6f25f392ebd934f9fa89d07ddb5b2ffa976c3422bc60f8c4ddaca5397ad/userdata/shm DeviceMajor:0 DeviceMinor:281 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48604fe3-9728-461e-ae37-8208b424e149/volumes/kubernetes.io~projected/kube-api-access-zxvzg DeviceMajor:0 DeviceMinor:234 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0de4046-bfed-4695-af0e-1ba806e6b031/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:360 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b57cc991-d758-435f-9238-0dac97f10a93/volumes/kubernetes.io~projected/kube-api-access-sl96f DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b25bd7be-290a-499a-a81d-c005793eace2/volumes/kubernetes.io~projected/kube-api-access-8c6wq DeviceMajor:0 DeviceMinor:240 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2555270f-a2ea-4d11-8dff-d130f9786e3d/volumes/kubernetes.io~projected/kube-api-access-q5lg2 DeviceMajor:0 DeviceMinor:242 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a79a11b92c5252da0a40c17154db0cf7ca937cde26f1e2192e5143f2bd2c2b6c/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80c2f3b680f02615b275e52c25d8dca0b759e27c479f7ef19bc486e6d75cd70b/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4425c17-5cc6-4afc-922c-d74694805196/volumes/kubernetes.io~projected/kube-api-access-8n62b DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c/volumes/kubernetes.io~projected/kube-api-access-hq5lq DeviceMajor:0 DeviceMinor:283 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d0de4046-bfed-4695-af0e-1ba806e6b031/volumes/kubernetes.io~projected/kube-api-access-dkrns DeviceMajor:0 DeviceMinor:361 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9260d15b-9772-4cb4-8588-8e82f89f39b8/volumes/kubernetes.io~projected/kube-api-access-hhg5z DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f30f499a-7d25-4ea7-aa1e-357ccbf0485f/volumes/kubernetes.io~projected/kube-api-access-qqqkf DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d0ceec34113f5a8a9eb64e5c5e2899b435a56fa1464f3bb1c6292ddbff3a336/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d6e9127f4b7588b09b87bdcb7399d252e4fa32a62cfd57756e7a7246766a2d3/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a6655a6-9079-4927-8e6c-09d075de438c/volumes/kubernetes.io~projected/kube-api-access-btsvv DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/68c010f0f76f47a515f05930ada535e464568c6ac669fdd14e110b17f7a77c3a/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fcd57864758c6e788a88dea6c88fb474427145eb04c28fd86dfce47b14f6d3b0/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/108d9224-ccd3-4925-a65b-0d59e938ff72/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/108d9224-ccd3-4925-a65b-0d59e938ff72/volumes/kubernetes.io~projected/kube-api-access-cg5wd DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/354d7018-887a-497f-b74f-195c8c88e184/volumes/kubernetes.io~projected/kube-api-access-fg6dt DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9c546cea-b474-44cf-84a5-b89784bda1d6/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:235 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1f69404f3c50f789279e5a3757d9a8b1b4787b5e39d06df736c090657cfed25/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ecfcc08a256f4626dbbeccb2c8ed8bb53b8fde82e97487eb2f786119c8230c2/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/108d9224-ccd3-4925-a65b-0d59e938ff72/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0c9e32359e3ef25 MacAddress:4a:48:03:3c:69:47 Speed:10000 Mtu:8901} {Name:16c546e51f74fc9 MacAddress:6a:6b:26:23:ca:25 Speed:10000 Mtu:8901} {Name:1b1cabad9b401ae MacAddress:f2:1d:16:92:d3:a8 Speed:10000 Mtu:8901} {Name:1d56f450ef97694 MacAddress:86:00:a7:3f:20:e0 Speed:10000 Mtu:8901} {Name:3b60f291a818eea MacAddress:2a:ce:a8:f1:6d:18 Speed:10000 Mtu:8901} {Name:3c243069849fdbd MacAddress:d2:12:21:bb:83:e3 Speed:10000 Mtu:8901} {Name:47b4e6f25f392eb MacAddress:ee:1c:18:c3:dd:2c Speed:10000 Mtu:8901} {Name:5ecfcc08a256f46 MacAddress:9a:1c:0b:c6:e7:1e Speed:10000 Mtu:8901} {Name:68c010f0f76f47a MacAddress:3e:36:d3:c6:d8:56 Speed:10000 Mtu:8901} {Name:6c6141e7f39691e MacAddress:72:6c:7a:05:df:3d Speed:10000 Mtu:8901} {Name:99c457ae3480cd3 MacAddress:a2:10:b3:2b:e5:fd Speed:10000 Mtu:8901} {Name:9d6e9127f4b7588 MacAddress:92:46:8c:64:ab:99 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:ec:62:96:59 Speed:0 Mtu:9001} {Name:br-int MacAddress:92:a0:9d:22:16:b1 Speed:0 Mtu:8901} {Name:c1d497aa54f9d2b MacAddress:4a:fc:17:ba:0d:81 Speed:10000 Mtu:8901} {Name:e59b0b9e9676220 MacAddress:66:91:1e:02:29:b6 Speed:10000 Mtu:8901} {Name:ebe7cc246a69488 MacAddress:66:8e:ba:b3:d2:85 Speed:10000 Mtu:8901} {Name:ee5c4e58b1d36c1 MacAddress:c6:46:f8:0a:51:2e Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:ec:62:96:59 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:f6:75:2e:bf:b4:40 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:9e:cc:f4:24:e1:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 01:51:13.048108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.045070    5005 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 01:51:13.048108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.045129    5005 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 01:51:13.048108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.045448    5005 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 01:51:13.048108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.045472    5005 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-26.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 01:51:13.048108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.045649    5005 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 01:51:13.048108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.045662    5005 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 01:51:13.048108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.045681    5005 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 01:51:13.048108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.045724    5005 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 01:51:13.048108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.045752    5005 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 01:51:13.048108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.045958    5005 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 01:51:13.048108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.046092    5005 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 01:51:13.048108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.046141    5005 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 01:51:13.048108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.046162    5005 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 01:51:13.048108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.046175    5005 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 01:51:13.048108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.046189    5005 kubelet.go:405] "Adding apiserver pod source"
Nov 03 01:51:13.048108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.046200    5005 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 01:51:13.048108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.047133    5005 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 01:51:13.048108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.047147    5005 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 01:51:13.048108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.047725    5005 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 01:51:13.059170 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.059152    5005 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 01:51:13.059170 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.059168    5005 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 01:51:13.059281 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.059271    5005 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 01:51:13.059475 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.059464    5005 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 01:51:13.059504 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.059493    5005 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 01:51:13.059587 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.059578    5005 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 01:51:13.059617 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.059594    5005 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 01:51:13.059617 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.059604    5005 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 01:51:13.059617 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.059612    5005 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 01:51:13.059706 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.059619    5005 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 01:51:13.059706 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.059625    5005 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 01:51:13.059706 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.059630    5005 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 01:51:13.059706 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.059636    5005 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 01:51:13.059706 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.059643    5005 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 01:51:13.059706 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.059648    5005 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 01:51:13.059706 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.059666    5005 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 01:51:13.059706 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.059678    5005 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 01:51:13.059929 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.059722    5005 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 01:51:13.059929 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.059733    5005 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 01:51:13.063407 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.063386    5005 server.go:1263] "Started kubelet"
Nov 03 01:51:13.063496 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.063461    5005 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 01:51:13.065059 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.063486    5005 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 01:51:13.065059 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.063591    5005 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 01:51:13.065059 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.063904    5005 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 01:51:13.063990 ip-10-0-139-26 systemd[1]: Started Kubernetes Kubelet.
Nov 03 01:51:13.066431 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.066410    5005 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 01:51:13.069343 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.069324    5005 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 01:51:13.069434 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.069344    5005 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 01:51:13.069434 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.069400    5005 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 01:45:31 +0000 UTC" deadline="2028-04-30 20:52:15.834503381 +0000 UTC"
Nov 03 01:51:13.069434 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.069415    5005 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13075h1m2.765090583s"
Nov 03 01:51:13.069654 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.069435    5005 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 01:51:13.069654 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.069448    5005 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 01:51:13.069654 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:13.069459    5005 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-26.us-west-2.compute.internal\" not found"
Nov 03 01:51:13.069654 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.069537    5005 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 01:51:13.071915 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.071898    5005 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 01:51:13.071915 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.071918    5005 factory.go:55] Registering systemd factory
Nov 03 01:51:13.072043 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.071929    5005 factory.go:223] Registration of the systemd container factory successfully
Nov 03 01:51:13.072207 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.072188    5005 factory.go:153] Registering CRI-O factory
Nov 03 01:51:13.072207 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.072207    5005 factory.go:223] Registration of the crio container factory successfully
Nov 03 01:51:13.072308 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.072225    5005 factory.go:103] Registering Raw factory
Nov 03 01:51:13.072308 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.072245    5005 manager.go:1199] Started watching for new ooms in manager
Nov 03 01:51:13.072603 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.072590    5005 manager.go:319] Starting recovery of all containers
Nov 03 01:51:13.072940 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.072874    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="108d9224-ccd3-4925-a65b-0d59e938ff72" volumeName="kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-kube-api-access-cg5wd" seLinuxMountContext=""
Nov 03 01:51:13.072940 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.072914    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="354d7018-887a-497f-b74f-195c8c88e184" volumeName="kubernetes.io/configmap/354d7018-887a-497f-b74f-195c8c88e184-service-ca-bundle" seLinuxMountContext=""
Nov 03 01:51:13.072940 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.072931    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="354d7018-887a-497f-b74f-195c8c88e184" volumeName="kubernetes.io/secret/354d7018-887a-497f-b74f-195c8c88e184-serving-cert" seLinuxMountContext=""
Nov 03 01:51:13.073108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.072944    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42a56b4b-283d-4e63-bb6a-59c5d5df0c2c" volumeName="kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq" seLinuxMountContext=""
Nov 03 01:51:13.073108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.072957    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62c3ca06-b2a4-49d7-84db-1390277be2c8" volumeName="kubernetes.io/projected/62c3ca06-b2a4-49d7-84db-1390277be2c8-kube-api-access-z46t7" seLinuxMountContext=""
Nov 03 01:51:13.073108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.072970    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a9e57fd-a6e6-4ddb-aa08-ac1390403544" volumeName="kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-stats-auth" seLinuxMountContext=""
Nov 03 01:51:13.073108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.072982    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8684a429-234f-4537-917e-5b49509bbdb5" volumeName="kubernetes.io/empty-dir/8684a429-234f-4537-917e-5b49509bbdb5-tmp-dir" seLinuxMountContext=""
Nov 03 01:51:13.073108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.072994    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c546cea-b474-44cf-84a5-b89784bda1d6" volumeName="kubernetes.io/empty-dir/9c546cea-b474-44cf-84a5-b89784bda1d6-ca-trust-extracted" seLinuxMountContext=""
Nov 03 01:51:13.073108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073010    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c546cea-b474-44cf-84a5-b89784bda1d6" volumeName="kubernetes.io/secret/9c546cea-b474-44cf-84a5-b89784bda1d6-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 01:51:13.073108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073021    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b25bd7be-290a-499a-a81d-c005793eace2" volumeName="kubernetes.io/projected/b25bd7be-290a-499a-a81d-c005793eace2-kube-api-access-8c6wq" seLinuxMountContext=""
Nov 03 01:51:13.073108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073034    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b25bd7be-290a-499a-a81d-c005793eace2" volumeName="kubernetes.io/secret/b25bd7be-290a-499a-a81d-c005793eace2-cert" seLinuxMountContext=""
Nov 03 01:51:13.073108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073047    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be449663-2797-487f-8559-0a96c563ae3a" volumeName="kubernetes.io/projected/be449663-2797-487f-8559-0a96c563ae3a-kube-api-access-5z56p" seLinuxMountContext=""
Nov 03 01:51:13.073108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073060    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb8ca898-6f5f-4727-9d6d-d6aed65adfae" volumeName="kubernetes.io/projected/cb8ca898-6f5f-4727-9d6d-d6aed65adfae-kube-api-access-f76x5" seLinuxMountContext=""
Nov 03 01:51:13.073108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073075    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0de4046-bfed-4695-af0e-1ba806e6b031" volumeName="kubernetes.io/projected/d0de4046-bfed-4695-af0e-1ba806e6b031-kube-api-access-dkrns" seLinuxMountContext=""
Nov 03 01:51:13.073108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073091    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="108d9224-ccd3-4925-a65b-0d59e938ff72" volumeName="kubernetes.io/secret/108d9224-ccd3-4925-a65b-0d59e938ff72-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 01:51:13.073108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073103    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2555270f-a2ea-4d11-8dff-d130f9786e3d" volumeName="kubernetes.io/projected/2555270f-a2ea-4d11-8dff-d130f9786e3d-kube-api-access-q5lg2" seLinuxMountContext=""
Nov 03 01:51:13.073108 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073115    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="354d7018-887a-497f-b74f-195c8c88e184" volumeName="kubernetes.io/empty-dir/354d7018-887a-497f-b74f-195c8c88e184-snapshots" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073128    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c546cea-b474-44cf-84a5-b89784bda1d6" volumeName="kubernetes.io/configmap/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-certificates" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073141    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c546cea-b474-44cf-84a5-b89784bda1d6" volumeName="kubernetes.io/secret/9c546cea-b474-44cf-84a5-b89784bda1d6-installation-pull-secrets" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073153    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd" volumeName="kubernetes.io/projected/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-kube-api-access-6t8q9" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073165    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0de4046-bfed-4695-af0e-1ba806e6b031" volumeName="kubernetes.io/configmap/d0de4046-bfed-4695-af0e-1ba806e6b031-signing-cabundle" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073177    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e49068d1-9ae4-47b3-9a07-3d695a0a2402" volumeName="kubernetes.io/configmap/e49068d1-9ae4-47b3-9a07-3d695a0a2402-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073189    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1ca62b-2da2-47a9-8e2a-34f2dc75a431" volumeName="kubernetes.io/empty-dir/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-tuned" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073202    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1ca62b-2da2-47a9-8e2a-34f2dc75a431" volumeName="kubernetes.io/projected/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-kube-api-access-f2d5x" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073215    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7a013c2-18a4-4372-adcb-8dfafa673f9c" volumeName="kubernetes.io/projected/c7a013c2-18a4-4372-adcb-8dfafa673f9c-kube-api-access-ltkx2" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073226    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e49068d1-9ae4-47b3-9a07-3d695a0a2402" volumeName="kubernetes.io/configmap/e49068d1-9ae4-47b3-9a07-3d695a0a2402-cni-binary-copy" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073238    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e49068d1-9ae4-47b3-9a07-3d695a0a2402" volumeName="kubernetes.io/projected/e49068d1-9ae4-47b3-9a07-3d695a0a2402-kube-api-access-jcpqh" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073250    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef90f36a-279e-4bfa-a97d-70132a266670" volumeName="kubernetes.io/configmap/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-config" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073267    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4425c17-5cc6-4afc-922c-d74694805196" volumeName="kubernetes.io/projected/f4425c17-5cc6-4afc-922c-d74694805196-kube-api-access-8n62b" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073281    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f0923bb-db25-4266-ad8d-2e1cfb7f602b" volumeName="kubernetes.io/secret/1f0923bb-db25-4266-ad8d-2e1cfb7f602b-agent-certs" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073294    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2555270f-a2ea-4d11-8dff-d130f9786e3d" volumeName="kubernetes.io/configmap/2555270f-a2ea-4d11-8dff-d130f9786e3d-config-volume" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073306    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ba28bb-4bab-4c0d-aa00-8b3bce675f23" volumeName="kubernetes.io/configmap/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-env-overrides" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073339    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62150808-f089-4723-aac5-d17d3a803abe" volumeName="kubernetes.io/projected/62150808-f089-4723-aac5-d17d3a803abe-kube-api-access-g5jp6" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073353    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baf5281f-5938-4417-b22f-2644034da7ef" volumeName="kubernetes.io/projected/baf5281f-5938-4417-b22f-2644034da7ef-kube-api-access-mc9gg" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073366    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e49068d1-9ae4-47b3-9a07-3d695a0a2402" volumeName="kubernetes.io/configmap/e49068d1-9ae4-47b3-9a07-3d695a0a2402-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073380    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15face97-689e-4fef-86c0-17a2a85829db" volumeName="kubernetes.io/configmap/15face97-689e-4fef-86c0-17a2a85829db-config" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073393    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2555270f-a2ea-4d11-8dff-d130f9786e3d" volumeName="kubernetes.io/empty-dir/2555270f-a2ea-4d11-8dff-d130f9786e3d-tmp-dir" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073405    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd" volumeName="kubernetes.io/secret/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-serving-cert" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073418    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="108d9224-ccd3-4925-a65b-0d59e938ff72" volumeName="kubernetes.io/configmap/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-certificates" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073430    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c546cea-b474-44cf-84a5-b89784bda1d6" volumeName="kubernetes.io/configmap/9c546cea-b474-44cf-84a5-b89784bda1d6-trusted-ca" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073444    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c546cea-b474-44cf-84a5-b89784bda1d6" volumeName="kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-bound-sa-token" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073457    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a7a1afb-11cc-4aa4-bae6-68ce495ebca8" volumeName="kubernetes.io/configmap/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-daemon-config" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073470    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="354d7018-887a-497f-b74f-195c8c88e184" volumeName="kubernetes.io/empty-dir/354d7018-887a-497f-b74f-195c8c88e184-tmp" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073496    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8684a429-234f-4537-917e-5b49509bbdb5" volumeName="kubernetes.io/projected/8684a429-234f-4537-917e-5b49509bbdb5-kube-api-access-h6lhl" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073509    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ba28bb-4bab-4c0d-aa00-8b3bce675f23" volumeName="kubernetes.io/configmap/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073522    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a9e57fd-a6e6-4ddb-aa08-ac1390403544" volumeName="kubernetes.io/projected/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-kube-api-access-rzfk9" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073537    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be449663-2797-487f-8559-0a96c563ae3a" volumeName="kubernetes.io/empty-dir/be449663-2797-487f-8559-0a96c563ae3a-data-volume" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073551    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62c3ca06-b2a4-49d7-84db-1390277be2c8" volumeName="kubernetes.io/configmap/62c3ca06-b2a4-49d7-84db-1390277be2c8-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073563    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9260d15b-9772-4cb4-8588-8e82f89f39b8" volumeName="kubernetes.io/projected/9260d15b-9772-4cb4-8588-8e82f89f39b8-kube-api-access-hhg5z" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073575    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd" volumeName="kubernetes.io/configmap/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-trusted-ca" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073587    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd" volumeName="kubernetes.io/configmap/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-config" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073600    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b57cc991-d758-435f-9238-0dac97f10a93" volumeName="kubernetes.io/configmap/b57cc991-d758-435f-9238-0dac97f10a93-iptables-alerter-script" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073612    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baf5281f-5938-4417-b22f-2644034da7ef" volumeName="kubernetes.io/configmap/baf5281f-5938-4417-b22f-2644034da7ef-config" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073625    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef90f36a-279e-4bfa-a97d-70132a266670" volumeName="kubernetes.io/projected/ef90f36a-279e-4bfa-a97d-70132a266670-kube-api-access-kkw2h" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073639    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c1ca62b-2da2-47a9-8e2a-34f2dc75a431" volumeName="kubernetes.io/empty-dir/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-tmp" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073651    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="108d9224-ccd3-4925-a65b-0d59e938ff72" volumeName="kubernetes.io/secret/108d9224-ccd3-4925-a65b-0d59e938ff72-installation-pull-secrets" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073674    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79fe02fa-99ce-4222-b569-6b323df47ab4" volumeName="kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073685    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c546cea-b474-44cf-84a5-b89784bda1d6" volumeName="kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-kube-api-access-85997" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073714    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b57cc991-d758-435f-9238-0dac97f10a93" volumeName="kubernetes.io/projected/b57cc991-d758-435f-9238-0dac97f10a93-kube-api-access-sl96f" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073728    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="108d9224-ccd3-4925-a65b-0d59e938ff72" volumeName="kubernetes.io/configmap/108d9224-ccd3-4925-a65b-0d59e938ff72-trusted-ca" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073742    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f0923bb-db25-4266-ad8d-2e1cfb7f602b" volumeName="kubernetes.io/configmap/1f0923bb-db25-4266-ad8d-2e1cfb7f602b-konnectivity-ca" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073754    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="354d7018-887a-497f-b74f-195c8c88e184" volumeName="kubernetes.io/configmap/354d7018-887a-497f-b74f-195c8c88e184-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073767    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="354d7018-887a-497f-b74f-195c8c88e184" volumeName="kubernetes.io/projected/354d7018-887a-497f-b74f-195c8c88e184-kube-api-access-fg6dt" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073778    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f30f499a-7d25-4ea7-aa1e-357ccbf0485f" volumeName="kubernetes.io/projected/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-kube-api-access-qqqkf" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073791    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15face97-689e-4fef-86c0-17a2a85829db" volumeName="kubernetes.io/secret/15face97-689e-4fef-86c0-17a2a85829db-serving-cert" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073805    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ba28bb-4bab-4c0d-aa00-8b3bce675f23" volumeName="kubernetes.io/configmap/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-ovnkube-config" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073817    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ba28bb-4bab-4c0d-aa00-8b3bce675f23" volumeName="kubernetes.io/projected/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-kube-api-access-flcc2" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073830    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a9e57fd-a6e6-4ddb-aa08-ac1390403544" volumeName="kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-default-certificate" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073843    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15face97-689e-4fef-86c0-17a2a85829db" volumeName="kubernetes.io/projected/15face97-689e-4fef-86c0-17a2a85829db-kube-api-access-zqsmb" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073856    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a7a1afb-11cc-4aa4-bae6-68ce495ebca8" volumeName="kubernetes.io/projected/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-kube-api-access-5h48h" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073869    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a6655a6-9079-4927-8e6c-09d075de438c" volumeName="kubernetes.io/projected/3a6655a6-9079-4927-8e6c-09d075de438c-kube-api-access-btsvv" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073885    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62c3ca06-b2a4-49d7-84db-1390277be2c8" volumeName="kubernetes.io/configmap/62c3ca06-b2a4-49d7-84db-1390277be2c8-service-ca-bundle" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073898    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62c3ca06-b2a4-49d7-84db-1390277be2c8" volumeName="kubernetes.io/empty-dir/62c3ca06-b2a4-49d7-84db-1390277be2c8-archives-path" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073910    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9260d15b-9772-4cb4-8588-8e82f89f39b8" volumeName="kubernetes.io/configmap/9260d15b-9772-4cb4-8588-8e82f89f39b8-serviceca" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073922    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baf5281f-5938-4417-b22f-2644034da7ef" volumeName="kubernetes.io/secret/baf5281f-5938-4417-b22f-2644034da7ef-serving-cert" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073935    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0de4046-bfed-4695-af0e-1ba806e6b031" volumeName="kubernetes.io/secret/d0de4046-bfed-4695-af0e-1ba806e6b031-signing-key" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073948    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a7a1afb-11cc-4aa4-bae6-68ce495ebca8" volumeName="kubernetes.io/configmap/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-cni-binary-copy" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073976    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38de413a-873a-4288-85f5-c453581a53d6" volumeName="kubernetes.io/projected/38de413a-873a-4288-85f5-c453581a53d6-kube-api-access-9ln9h" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.073991    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54ba28bb-4bab-4c0d-aa00-8b3bce675f23" volumeName="kubernetes.io/secret/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.074007    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="665f9d02-067e-4506-9793-cc7f17eb95bb" volumeName="kubernetes.io/projected/665f9d02-067e-4506-9793-cc7f17eb95bb-kube-api-access-cvkdd" seLinuxMountContext=""
Nov 03 01:51:13.073982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.074020    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4425c17-5cc6-4afc-922c-d74694805196" volumeName="kubernetes.io/configmap/f4425c17-5cc6-4afc-922c-d74694805196-telemetry-config" seLinuxMountContext=""
Nov 03 01:51:13.076813 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.074035    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="108d9224-ccd3-4925-a65b-0d59e938ff72" volumeName="kubernetes.io/empty-dir/108d9224-ccd3-4925-a65b-0d59e938ff72-ca-trust-extracted" seLinuxMountContext=""
Nov 03 01:51:13.076813 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.074048    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48604fe3-9728-461e-ae37-8208b424e149" volumeName="kubernetes.io/projected/48604fe3-9728-461e-ae37-8208b424e149-kube-api-access-zxvzg" seLinuxMountContext=""
Nov 03 01:51:13.076813 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.074062    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e11ea02d-ac52-4318-9fdd-95c0bb171aed" volumeName="kubernetes.io/projected/e11ea02d-ac52-4318-9fdd-95c0bb171aed-kube-api-access-2xbq7" seLinuxMountContext=""
Nov 03 01:51:13.076813 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.074075    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0110b28b-ec04-4f65-bc3d-2e82f88b30d7" volumeName="kubernetes.io/configmap/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-nginx-conf" seLinuxMountContext=""
Nov 03 01:51:13.076813 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.074087    5005 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="108d9224-ccd3-4925-a65b-0d59e938ff72" volumeName="kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-bound-sa-token" seLinuxMountContext=""
Nov 03 01:51:13.076813 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.074099    5005 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 01:51:13.076813 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.074106    5005 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 01:51:13.082714 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.078975    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/1.log"
Nov 03 01:51:13.082714 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.081604    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/1.log"
Nov 03 01:51:13.099709 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.099647    5005 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 01:51:13.101105 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.101079    5005 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 01:51:13.101105 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.101102    5005 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 01:51:13.101258 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.101118    5005 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 01:51:13.101258 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:13.101149    5005 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 01:51:13.119784 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.119753    5005 generic.go:356] "Generic (PLEG): container finished" podID="6076c2cc730e7ded3ca7127f0bb3e92b" containerID="dae4a6cea5adfe867282e761d33fd8007491853908768ea609e1e2c0915b0e1d" exitCode=0
Nov 03 01:51:13.132924 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.132903    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/1.log"
Nov 03 01:51:13.133019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.132928    5005 generic.go:356] "Generic (PLEG): container finished" podID="a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd" containerID="8df6474c221450b3bf6a0de2c5673ab2c142dbf0053634b6d05d2b7c199801e8" exitCode=255
Nov 03 01:51:13.153449 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.153423    5005 generic.go:356] "Generic (PLEG): container finished" podID="e49068d1-9ae4-47b3-9a07-3d695a0a2402" containerID="308252bb402cfd85e2daddb0bb79145bb24b19b75eeaf6c813430d3a7fe4fdc6" exitCode=0
Nov 03 01:51:13.153449 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.153448    5005 generic.go:356] "Generic (PLEG): container finished" podID="e49068d1-9ae4-47b3-9a07-3d695a0a2402" containerID="51604b80f0fd685dce59fddb01b34582be172c46bc9c4ee2a290ca691b1679d1" exitCode=0
Nov 03 01:51:13.153611 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.153460    5005 generic.go:356] "Generic (PLEG): container finished" podID="e49068d1-9ae4-47b3-9a07-3d695a0a2402" containerID="a681afd79e83f58433cfd20760f2c818d47c622035cd44a992bf3ccc6d9c1a9d" exitCode=0
Nov 03 01:51:13.153611 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.153466    5005 generic.go:356] "Generic (PLEG): container finished" podID="e49068d1-9ae4-47b3-9a07-3d695a0a2402" containerID="926b1dbf62a0a1fbe202cfb6da2b50d1d205e96072e6d0001cf1cc6807eec7fa" exitCode=0
Nov 03 01:51:13.153611 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.153471    5005 generic.go:356] "Generic (PLEG): container finished" podID="e49068d1-9ae4-47b3-9a07-3d695a0a2402" containerID="3e9a85060a97f5d0567360cd0401796581ebfa967dc634ce8662739a801c611a" exitCode=0
Nov 03 01:51:13.153611 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.153478    5005 generic.go:356] "Generic (PLEG): container finished" podID="e49068d1-9ae4-47b3-9a07-3d695a0a2402" containerID="a3869b735a7d1363920c3adf674317a711fc4c5836981f79765233b6631d3562" exitCode=0
Nov 03 01:51:13.164073 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.164055    5005 manager.go:324] Recovery completed
Nov 03 01:51:13.169562 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:13.169536    5005 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-26.us-west-2.compute.internal\" not found"
Nov 03 01:51:13.181669 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.181655    5005 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:51:13.183038 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.183022    5005 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:51:13.183112 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.183051    5005 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:51:13.183112 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.183065    5005 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:51:13.186100 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.186086    5005 cpu_manager.go:226] "Starting" policy="none"
Nov 03 01:51:13.186100 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.186098    5005 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 01:51:13.186201 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.186100    5005 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:51:13.186201 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.186114    5005 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 01:51:13.186289 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.186233    5005 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 01:51:13.186289 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.186247    5005 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 01:51:13.186289 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.186268    5005 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 01:51:13.186289 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.186275    5005 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 01:51:13.186289 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.186283    5005 policy_none.go:50] "Start"
Nov 03 01:51:13.186423 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.186294    5005 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 01:51:13.186423 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.186304    5005 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 01:51:13.186477 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.186433    5005 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 01:51:13.186477 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.186442    5005 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 01:51:13.186477 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.186449    5005 policy_none.go:44] "Start"
Nov 03 01:51:13.187022 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.187006    5005 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:51:13.187072 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.187035    5005 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:51:13.187072 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.187045    5005 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:51:13.192310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.192283    5005 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 01:51:13.195600 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.195589    5005 manager.go:341] "Starting Device Plugin manager"
Nov 03 01:51:13.195676 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:13.195617    5005 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 01:51:13.195676 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.195628    5005 server.go:87] "Starting device plugin registration server"
Nov 03 01:51:13.195893 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.195882    5005 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 01:51:13.195926 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.195894    5005 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 01:51:13.195995 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.195980    5005 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 01:51:13.196120 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.196086    5005 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 01:51:13.196226 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.196125    5005 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 01:51:13.196226 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.196132    5005 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 01:51:13.196945 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.196839    5005 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T01:51:13.196115074Z","UUID":"8d8c5bed-3f7b-4600-9cfe-f5ce1a6be853","Handler":null,"Name":"","Endpoint":""}
Nov 03 01:51:13.198120 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:13.198101    5005 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 01:51:13.198218 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.198133    5005 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 01:51:13.198218 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.198153    5005 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 01:51:13.198218 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:13.198136    5005 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-26.us-west-2.compute.internal\" not found"
Nov 03 01:51:13.201236 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.201198    5005 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal"]
Nov 03 01:51:13.201363 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.201351    5005 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:51:13.203055 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.203039    5005 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:51:13.203123 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.203070    5005 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:51:13.203123 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.203086    5005 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:51:13.203501 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.203452    5005 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:51:13.203605 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.203589    5005 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:51:13.204320 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.204304    5005 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:51:13.204320 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.204317    5005 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:51:13.204438 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.204330    5005 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:51:13.204438 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.204341    5005 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:51:13.204438 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.204341    5005 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:51:13.204438 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.204377    5005 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:51:13.204589 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:13.204468    5005 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-26.us-west-2.compute.internal\" not found" node="ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:13.204739 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.204685    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal" event={"ID":"6076c2cc730e7ded3ca7127f0bb3e92b","Type":"ContainerStarted","Data":"5d829ef3c1903ffa6e014a1b0b39feb49170a1a6a528261a80b5c984af7b193a"}
Nov 03 01:51:13.204794 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.204753    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal" event={"ID":"6076c2cc730e7ded3ca7127f0bb3e92b","Type":"ContainerDied","Data":"dae4a6cea5adfe867282e761d33fd8007491853908768ea609e1e2c0915b0e1d"}
Nov 03 01:51:13.204794 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.204771    5005 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:51:13.204879 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.204773    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal" event={"ID":"6076c2cc730e7ded3ca7127f0bb3e92b","Type":"ContainerStarted","Data":"8a2b0a911b8a92222130cb200c593379e9a7975d1527e41c6b12a12dae6af66e"}
Nov 03 01:51:13.205001 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.204984    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal" event={"ID":"37711de5beea2605bd7d0ac9a465c416","Type":"ContainerStarted","Data":"9973f3a32c179e2ebb266c3b38926cebea8e27de5aa01f7e62f64e7999d4f934"}
Nov 03 01:51:13.205053 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.205015    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal" event={"ID":"37711de5beea2605bd7d0ac9a465c416","Type":"ContainerStarted","Data":"fcd57864758c6e788a88dea6c88fb474427145eb04c28fd86dfce47b14f6d3b0"}
Nov 03 01:51:13.205551 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.205535    5005 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:51:13.205606 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.205560    5005 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:51:13.205606 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.205569    5005 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:51:13.205739 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:13.205724    5005 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-26.us-west-2.compute.internal\" not found" node="ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:13.240856 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.240831    5005 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 01:51:13.240966 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.240953    5005 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 01:51:13.241874 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.241855    5005 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 01:51:13.241957 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.241943    5005 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 01:51:13.270002 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.269975    5005 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:13.288601 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.288575    5005 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 01:51:13.296824 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.296806    5005 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:51:13.297985 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.297967    5005 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:51:13.298060 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.298004    5005 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:51:13.298060 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.298020    5005 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:51:13.298118 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.298071    5005 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:13.326966 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.326938    5005 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 01:51:13.327053 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:13.326987    5005 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:13.327053 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.327000    5005 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:13.341464 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.341437    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6076c2cc730e7ded3ca7127f0bb3e92b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal\" (UID: \"6076c2cc730e7ded3ca7127f0bb3e92b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:13.341525 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.341475    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6076c2cc730e7ded3ca7127f0bb3e92b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal\" (UID: \"6076c2cc730e7ded3ca7127f0bb3e92b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:13.341525 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.341504    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37711de5beea2605bd7d0ac9a465c416-config\") pod \"kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal\" (UID: \"37711de5beea2605bd7d0ac9a465c416\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:13.399462 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.399436    5005 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:13.399577 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.399501    5005 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:13.426324 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.426303    5005 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 01:51:13.426415 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:13.426340    5005 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:13.441930 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.441908    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6076c2cc730e7ded3ca7127f0bb3e92b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal\" (UID: \"6076c2cc730e7ded3ca7127f0bb3e92b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:13.442016 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.441934    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6076c2cc730e7ded3ca7127f0bb3e92b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal\" (UID: \"6076c2cc730e7ded3ca7127f0bb3e92b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:13.442016 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.441952    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37711de5beea2605bd7d0ac9a465c416-config\") pod \"kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal\" (UID: \"37711de5beea2605bd7d0ac9a465c416\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:13.442016 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.441989    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37711de5beea2605bd7d0ac9a465c416-config\") pod \"kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal\" (UID: \"37711de5beea2605bd7d0ac9a465c416\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:13.442016 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.442007    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6076c2cc730e7ded3ca7127f0bb3e92b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal\" (UID: \"6076c2cc730e7ded3ca7127f0bb3e92b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:13.442136 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:13.442007    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6076c2cc730e7ded3ca7127f0bb3e92b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal\" (UID: \"6076c2cc730e7ded3ca7127f0bb3e92b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:14.047341 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.047303    5005 apiserver.go:52] "Watching apiserver"
Nov 03 01:51:14.158064 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.158035    5005 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:14.158226 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.158163    5005 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:14.207797 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.207771    5005 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 01:51:14.208472 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.208445    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr","openshift-dns/dns-default-w4h82","openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8","openshift-network-console/networking-console-plugin-86c478bb88-55bnn","openshift-ingress-canary/ingress-canary-7t99k","openshift-ovn-kubernetes/ovnkube-node-jdk59","openshift-insights/insights-runtime-extractor-9twrb","kube-system/global-pull-secret-syncer-mxg52","openshift-image-registry/node-ca-qg2kq","openshift-ingress/router-default-7875dd7849-ptjkr","kube-system/kas-connection-checker-685895c7c9-l87nq","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh","openshift-kube-storage-version-migrator/migrator-66df5d496b-4wq24","openshift-multus/multus-9kq9k","openshift-insights/periodic-gathering-gxg2j-p5824","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-f5dm4","kube-system/konnectivity-agent-vvzws","openshift-dns/node-resolver-p8kgq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-mdr8p","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn","openshift-console-operator/console-operator-7cd56b8dbc-rjmbd","openshift-network-operator/iptables-alerter-tp5hd","openshift-kube-storage-version-migrator/migrator-66df5d496b-t9rqp","openshift-service-ca/service-ca-5fffc97bdf-qlt9s","kube-system/kas-connection-checker-685895c7c9-rcv8q","kube-system/kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal","kube-system/kas-connection-checker-685895c7c9-jqkfr","openshift-image-registry/image-registry-55fd5cff67-bb76l","openshift-cluster-node-tuning-operator/tuned-25wbg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs","openshift-image-registry/image-registry-6bcfc5667d-w8v7m","openshift-multus/multus-additional-cni-plugins-d7qsp","openshift-insights/insights-operator-7c4986d449-gqz7v","openshift-network-diagnostics/network-check-source-6f78d59f99-z28hp","openshift-multus/network-metrics-daemon-xv8zr"]
Nov 03 01:51:14.209577 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.209556    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:14.209910 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.209894    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:14.211502 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.211482    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:14.212685 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.212669    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn"
Nov 03 01:51:14.213127 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.213113    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:14.213991 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.213974    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:51:14.215149 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.215133    5005 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 01:51:14.218001 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.215525    5005 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 01:51:14.223764 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.223747    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:14.224287 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.224276    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:14.227044 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.227027    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:14.253083 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.253061    5005 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:14.253083 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.253077    5005 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal"
Nov 03 01:51:14.258184 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.258168    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 01:51:14.258264 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.258238    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 01:51:14.258544 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.258525    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 01:51:14.258635 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.258556    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 01:51:14.258635 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.258632    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 01:51:14.258740 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.258661    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 01:51:14.258740 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.258738    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-k78r9\""
Nov 03 01:51:14.258817 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.258738    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-54dhc\""
Nov 03 01:51:14.258952 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.258937    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 01:51:14.258985 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.258963    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 01:51:14.259021 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.259009    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 01:51:14.259613 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.259598    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rwql9\""
Nov 03 01:51:14.259744 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.259729    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 01:51:14.259786 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.259770    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 01:51:14.259815 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.259783    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mlvnc\""
Nov 03 01:51:14.259858 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.259813    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jfs9g\""
Nov 03 01:51:14.259858 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.259819    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-87fvc\""
Nov 03 01:51:14.260239 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.260225    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-m9xwl\""
Nov 03 01:51:14.260529 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.260515    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 01:51:14.260582 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.260529    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 01:51:14.260582 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.260537    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 01:51:14.260582 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.260571    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 01:51:14.260881 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.260864    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 01:51:14.260946 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.260881    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9tjtm\""
Nov 03 01:51:14.260946 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.260899    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 01:51:14.260946 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.260904    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 01:51:14.261050 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.260971    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-kgn4w\""
Nov 03 01:51:14.261050 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.261042    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 01:51:14.261106 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.261054    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 01:51:14.261106 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.261046    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 01:51:14.261368 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.261353    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-hc9w6\""
Nov 03 01:51:14.261734 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.261719    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bnxtn\""
Nov 03 01:51:14.262021 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.262002    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 01:51:14.262102 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.262051    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 01:51:14.262155 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.262112    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 01:51:14.262213 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.262202    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 01:51:14.262246 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.262206    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 01:51:14.262246 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.262212    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 01:51:14.262323 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.262272    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 01:51:14.262323 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.262277    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 01:51:14.262323 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.262313    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 01:51:14.262461 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.262349    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 01:51:14.262461 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.262367    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 01:51:14.262461 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.262376    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 01:51:14.262461 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.262383    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 01:51:14.262461 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.262386    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 01:51:14.262461 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.262417    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 01:51:14.262630 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.262487    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 01:51:14.272382 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.272358    5005 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 01:51:14.303433 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.303385    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 01:51:14.307457 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.307439    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 01:51:14.307532 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.307473    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 01:51:14.307587 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.307534    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 01:51:14.307587 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.307569    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 01:51:14.307718 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.307583    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 01:51:14.307718 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.307625    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 01:51:14.307919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.307730    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 01:51:14.307919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.307756    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 01:51:14.307919 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.307766    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 01:51:14.308082 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.307925    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 01:51:14.308082 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308039    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 01:51:14.308194 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308122    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 01:51:14.308194 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308129    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 01:51:14.308194 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308147    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 01:51:14.308194 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308156    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 01:51:14.308361 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308201    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 01:51:14.308361 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308232    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 01:51:14.308361 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308248    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 01:51:14.308361 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308260    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-btxgv\""
Nov 03 01:51:14.308361 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308318    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 01:51:14.308565 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308382    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 01:51:14.308565 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308458    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 01:51:14.308565 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308491    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 01:51:14.308565 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308514    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 01:51:14.308748 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308609    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 01:51:14.308748 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308609    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 01:51:14.308748 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308658    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 01:51:14.308847 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308756    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zq5dk\""
Nov 03 01:51:14.308847 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308759    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 01:51:14.308847 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308785    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 01:51:14.308847 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308792    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 01:51:14.308847 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308800    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 01:51:14.308847 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.308844    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 01:51:14.309258 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.309239    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 01:51:14.309360 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.309342    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 01:51:14.309425 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.309375    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jwmm7\""
Nov 03 01:51:14.309425 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.309399    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2mmqt\""
Nov 03 01:51:14.309425 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.309406    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zxsvp\""
Nov 03 01:51:14.309425 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.309378    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-h8v6w\""
Nov 03 01:51:14.309425 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.309400    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 01:51:14.309603 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.309427    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 01:51:14.309603 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.309449    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pc8bh\""
Nov 03 01:51:14.309603 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.309457    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5qdp4\""
Nov 03 01:51:14.309603 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.309487    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f7c55\""
Nov 03 01:51:14.309877 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.309860    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 01:51:14.309970 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.309952    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hgghc\""
Nov 03 01:51:14.310042 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.309994    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 01:51:14.310042 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.310016    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 01:51:14.310162 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.309476    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-d7skx\""
Nov 03 01:51:14.310302 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.310288    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 01:51:14.310386 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.310333    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7cgvv\""
Nov 03 01:51:14.310488 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.310409    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 01:51:14.310719 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.310681    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 01:51:14.311037 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.311020    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 01:51:14.311121 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.311060    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-v5ks7\""
Nov 03 01:51:14.311121 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.311077    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wpc48\""
Nov 03 01:51:14.311233 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.311197    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bsx6s\""
Nov 03 01:51:14.311413 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.311391    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 01:51:14.311570 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.311538    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jwnvd\""
Nov 03 01:51:14.311652 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.311630    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bqcwb\""
Nov 03 01:51:14.312040 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.312013    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 01:51:14.346205 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.346181    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-kubelet\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.346286 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.346208    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-cni-bin\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.346286 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.346224    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be449663-2797-487f-8559-0a96c563ae3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:14.346286 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.346251    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxvzg\" (UniqueName: \"kubernetes.io/projected/48604fe3-9728-461e-ae37-8208b424e149-kube-api-access-zxvzg\") pod \"volume-data-source-validator-58c9f5f46d-f5dm4\" (UID: \"48604fe3-9728-461e-ae37-8208b424e149\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-f5dm4"
Nov 03 01:51:14.346423 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.346316    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/108d9224-ccd3-4925-a65b-0d59e938ff72-installation-pull-secrets\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:14.346423 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.346355    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-run\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.346513 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.346453    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-host\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.346513 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.346501    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b57cc991-d758-435f-9238-0dac97f10a93-host-slash\") pod \"iptables-alerter-tp5hd\" (UID: \"b57cc991-d758-435f-9238-0dac97f10a93\") " pod="openshift-network-operator/iptables-alerter-tp5hd"
Nov 03 01:51:14.346590 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.346527    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/108d9224-ccd3-4925-a65b-0d59e938ff72-installation-pull-secrets\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:14.346590 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.346535    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z46t7\" (UniqueName: \"kubernetes.io/projected/62c3ca06-b2a4-49d7-84db-1390277be2c8-kube-api-access-z46t7\") pod \"periodic-gathering-gxg2j-p5824\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") " pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:51:14.346664 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.346649    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg5wd\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-kube-api-access-cg5wd\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:14.346723 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.346685    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-lib-modules\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.346762 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.346739    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ln9h\" (UniqueName: \"kubernetes.io/projected/38de413a-873a-4288-85f5-c453581a53d6-kube-api-access-9ln9h\") pod \"kas-connection-checker-685895c7c9-l87nq\" (UID: \"38de413a-873a-4288-85f5-c453581a53d6\") " pod="kube-system/kas-connection-checker-685895c7c9-l87nq"
Nov 03 01:51:14.346810 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.346775    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79fe02fa-99ce-4222-b569-6b323df47ab4-kubelet-config\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:51:14.346884 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.346866    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/354d7018-887a-497f-b74f-195c8c88e184-tmp\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:14.346935 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.346890    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e49068d1-9ae4-47b3-9a07-3d695a0a2402-system-cni-dir\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:51:14.346935 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.346907    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2555270f-a2ea-4d11-8dff-d130f9786e3d-tmp-dir\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:14.347009 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.346941    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79fe02fa-99ce-4222-b569-6b323df47ab4-dbus\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:51:14.347009 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.346962    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:14.347009 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.346980    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-device-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:51:14.347009 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.346990    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/354d7018-887a-497f-b74f-195c8c88e184-tmp\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:14.347185 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347031    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2555270f-a2ea-4d11-8dff-d130f9786e3d-tmp-dir\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:14.347185 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347033    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:51:14.347185 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347079    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:14.347185 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347124    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/baf5281f-5938-4417-b22f-2644034da7ef-config\") pod \"service-ca-operator-544d749c8c-nw7sr\" (UID: \"baf5281f-5938-4417-b22f-2644034da7ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr"
Nov 03 01:51:14.347185 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347149    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h48h\" (UniqueName: \"kubernetes.io/projected/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-kube-api-access-5h48h\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.347185 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347177    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.347431 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347204    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-config\") pod \"console-operator-7cd56b8dbc-rjmbd\" (UID: \"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:14.347431 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347228    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9260d15b-9772-4cb4-8588-8e82f89f39b8-host\") pod \"node-ca-qg2kq\" (UID: \"9260d15b-9772-4cb4-8588-8e82f89f39b8\") " pod="openshift-image-registry/node-ca-qg2kq"
Nov 03 01:51:14.347431 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347250    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-system-cni-dir\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.347431 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347261    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/baf5281f-5938-4417-b22f-2644034da7ef-config\") pod \"service-ca-operator-544d749c8c-nw7sr\" (UID: \"baf5281f-5938-4417-b22f-2644034da7ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr"
Nov 03 01:51:14.347431 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347273    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-cnibin\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.347431 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347311    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-slash\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.347431 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347370    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-etc-openvswitch\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.347431 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347411    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-node-log\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.347868 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347439    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:14.347868 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347451    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-config\") pod \"console-operator-7cd56b8dbc-rjmbd\" (UID: \"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:14.347868 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347467    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-certificates\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:14.347868 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347493    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:51:14.347868 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347520    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-bound-sa-token\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:14.347868 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347541    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-cni-binary-copy\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.347868 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347558    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n62b\" (UniqueName: \"kubernetes.io/projected/f4425c17-5cc6-4afc-922c-d74694805196-kube-api-access-8n62b\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:14.347868 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347576    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-sys\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.347868 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347659    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-var-lib-kubelet\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.347868 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347632    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-certificates\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:14.347868 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347715    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e49068d1-9ae4-47b3-9a07-3d695a0a2402-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:51:14.347868 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347750    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq5lq\" (UniqueName: \"kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq\") pod \"network-check-target-mdr8p\" (UID: \"42a56b4b-283d-4e63-bb6a-59c5d5df0c2c\") " pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:51:14.347868 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347798    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8684a429-234f-4537-917e-5b49509bbdb5-tmp-dir\") pod \"node-resolver-p8kgq\" (UID: \"8684a429-234f-4537-917e-5b49509bbdb5\") " pod="openshift-dns/node-resolver-p8kgq"
Nov 03 01:51:14.347868 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347851    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79fe02fa-99ce-4222-b569-6b323df47ab4-original-pull-secret\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:51:14.348437 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347883    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c546cea-b474-44cf-84a5-b89784bda1d6-trusted-ca\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:14.348437 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347923    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8684a429-234f-4537-917e-5b49509bbdb5-tmp-dir\") pod \"node-resolver-p8kgq\" (UID: \"8684a429-234f-4537-917e-5b49509bbdb5\") " pod="openshift-dns/node-resolver-p8kgq"
Nov 03 01:51:14.348437 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347965    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-run-multus-certs\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.348437 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.347994    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-ovnkube-script-lib\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.348437 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348026    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e49068d1-9ae4-47b3-9a07-3d695a0a2402-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:51:14.348437 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348076    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltkx2\" (UniqueName: \"kubernetes.io/projected/c7a013c2-18a4-4372-adcb-8dfafa673f9c-kube-api-access-ltkx2\") pod \"network-metrics-daemon-xv8zr\" (UID: \"c7a013c2-18a4-4372-adcb-8dfafa673f9c\") " pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:51:14.348437 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348123    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-cni-binary-copy\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.348437 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348170    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-run-ovn-kubernetes\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.348437 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348242    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b57cc991-d758-435f-9238-0dac97f10a93-iptables-alerter-script\") pod \"iptables-alerter-tp5hd\" (UID: \"b57cc991-d758-435f-9238-0dac97f10a93\") " pod="openshift-network-operator/iptables-alerter-tp5hd"
Nov 03 01:51:14.348437 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348274    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c546cea-b474-44cf-84a5-b89784bda1d6-trusted-ca\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:14.348437 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348273    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl96f\" (UniqueName: \"kubernetes.io/projected/b57cc991-d758-435f-9238-0dac97f10a93-kube-api-access-sl96f\") pod \"iptables-alerter-tp5hd\" (UID: \"b57cc991-d758-435f-9238-0dac97f10a93\") " pod="openshift-network-operator/iptables-alerter-tp5hd"
Nov 03 01:51:14.348437 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348308    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/baf5281f-5938-4417-b22f-2644034da7ef-serving-cert\") pod \"service-ca-operator-544d749c8c-nw7sr\" (UID: \"baf5281f-5938-4417-b22f-2644034da7ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr"
Nov 03 01:51:14.348437 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348317    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-ovnkube-script-lib\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.348437 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348330    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-kubernetes\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.348437 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348353    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-var-lib-kubelet\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.348437 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348379    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xbq7\" (UniqueName: \"kubernetes.io/projected/e11ea02d-ac52-4318-9fdd-95c0bb171aed-kube-api-access-2xbq7\") pod \"kas-connection-checker-685895c7c9-jqkfr\" (UID: \"e11ea02d-ac52-4318-9fdd-95c0bb171aed\") " pod="kube-system/kas-connection-checker-685895c7c9-jqkfr"
Nov 03 01:51:14.348437 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348425    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-55bnn\" (UID: \"0110b28b-ec04-4f65-bc3d-2e82f88b30d7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn"
Nov 03 01:51:14.348437 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348443    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/baf5281f-5938-4417-b22f-2644034da7ef-serving-cert\") pod \"service-ca-operator-544d749c8c-nw7sr\" (UID: \"baf5281f-5938-4417-b22f-2644034da7ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348479    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqsmb\" (UniqueName: \"kubernetes.io/projected/15face97-689e-4fef-86c0-17a2a85829db-kube-api-access-zqsmb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t59nn\" (UID: \"15face97-689e-4fef-86c0-17a2a85829db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348504    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c546cea-b474-44cf-84a5-b89784bda1d6-installation-pull-secrets\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348522    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-var-lib-cni-multus\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348537    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e49068d1-9ae4-47b3-9a07-3d695a0a2402-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348556    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-serving-cert\") pod \"console-operator-7cd56b8dbc-rjmbd\" (UID: \"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348582    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be449663-2797-487f-8559-0a96c563ae3a-data-volume\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348598    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348626    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62c3ca06-b2a4-49d7-84db-1390277be2c8-trusted-ca-bundle\") pod \"periodic-gathering-gxg2j-p5824\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") " pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348635    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c546cea-b474-44cf-84a5-b89784bda1d6-installation-pull-secrets\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348658    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/15face97-689e-4fef-86c0-17a2a85829db-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t59nn\" (UID: \"15face97-689e-4fef-86c0-17a2a85829db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348681    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-sysctl-d\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348688    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be449663-2797-487f-8559-0a96c563ae3a-data-volume\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348724    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e49068d1-9ae4-47b3-9a07-3d695a0a2402-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348729    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-var-lib-cni-bin\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348765    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-serving-cert\") pod \"console-operator-7cd56b8dbc-rjmbd\" (UID: \"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348760    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-cni-netd\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348805    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/15face97-689e-4fef-86c0-17a2a85829db-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t59nn\" (UID: \"15face97-689e-4fef-86c0-17a2a85829db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348847    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/108d9224-ccd3-4925-a65b-0d59e938ff72-trusted-ca\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348893    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-modprobe-d\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348918    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/15face97-689e-4fef-86c0-17a2a85829db-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t59nn\" (UID: \"15face97-689e-4fef-86c0-17a2a85829db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348919    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-sysctl-conf\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348956    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62c3ca06-b2a4-49d7-84db-1390277be2c8-trusted-ca-bundle\") pod \"periodic-gathering-gxg2j-p5824\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") " pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348963    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-conf-dir\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348982    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/15face97-689e-4fef-86c0-17a2a85829db-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t59nn\" (UID: \"15face97-689e-4fef-86c0-17a2a85829db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.348988    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5jp6\" (UniqueName: \"kubernetes.io/projected/62150808-f089-4723-aac5-d17d3a803abe-kube-api-access-g5jp6\") pod \"migrator-66df5d496b-t9rqp\" (UID: \"62150808-f089-4723-aac5-d17d3a803abe\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t9rqp"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349030    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b25bd7be-290a-499a-a81d-c005793eace2-cert\") pod \"ingress-canary-7t99k\" (UID: \"b25bd7be-290a-499a-a81d-c005793eace2\") " pod="openshift-ingress-canary/ingress-canary-7t99k"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349053    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-sysconfig\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349099    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/354d7018-887a-497f-b74f-195c8c88e184-serving-cert\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349111    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/108d9224-ccd3-4925-a65b-0d59e938ff72-trusted-ca\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349118    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-run-k8s-cni-cncf-io\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.349187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349200    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349214    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b25bd7be-290a-499a-a81d-c005793eace2-cert\") pod \"ingress-canary-7t99k\" (UID: \"b25bd7be-290a-499a-a81d-c005793eace2\") " pod="openshift-ingress-canary/ingress-canary-7t99k"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349228    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-qtpnh\" (UID: \"ef90f36a-279e-4bfa-a97d-70132a266670\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349243    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/354d7018-887a-497f-b74f-195c8c88e184-serving-cert\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349256    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/354d7018-887a-497f-b74f-195c8c88e184-service-ca-bundle\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349280    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-hostroot\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349383    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-env-overrides\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349405    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-qtpnh\" (UID: \"ef90f36a-279e-4bfa-a97d-70132a266670\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349437    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-trusted-ca\") pod \"console-operator-7cd56b8dbc-rjmbd\" (UID: \"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349446    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/354d7018-887a-497f-b74f-195c8c88e184-service-ca-bundle\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349462    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8684a429-234f-4537-917e-5b49509bbdb5-hosts-file\") pod \"node-resolver-p8kgq\" (UID: \"8684a429-234f-4537-917e-5b49509bbdb5\") " pod="openshift-dns/node-resolver-p8kgq"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349487    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-env-overrides\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349569    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f76x5\" (UniqueName: \"kubernetes.io/projected/cb8ca898-6f5f-4727-9d6d-d6aed65adfae-kube-api-access-f76x5\") pod \"migrator-66df5d496b-4wq24\" (UID: \"cb8ca898-6f5f-4727-9d6d-d6aed65adfae\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4wq24"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349586    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-socket-dir-parent\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349603    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs\") pod \"network-metrics-daemon-xv8zr\" (UID: \"c7a013c2-18a4-4372-adcb-8dfafa673f9c\") " pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349618    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349637    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-tuned\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349652    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6lhl\" (UniqueName: \"kubernetes.io/projected/8684a429-234f-4537-917e-5b49509bbdb5-kube-api-access-h6lhl\") pod \"node-resolver-p8kgq\" (UID: \"8684a429-234f-4537-917e-5b49509bbdb5\") " pod="openshift-dns/node-resolver-p8kgq"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349717    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-tuned\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349769    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-trusted-ca\") pod \"console-operator-7cd56b8dbc-rjmbd\" (UID: \"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349780    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-default-certificate\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349797    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d0de4046-bfed-4695-af0e-1ba806e6b031-signing-key\") pod \"service-ca-5fffc97bdf-qlt9s\" (UID: \"d0de4046-bfed-4695-af0e-1ba806e6b031\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlt9s"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349813    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-run-netns\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349846    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqkf\" (UniqueName: \"kubernetes.io/projected/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-kube-api-access-qqqkf\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349861    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e49068d1-9ae4-47b3-9a07-3d695a0a2402-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349879    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349900    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-tmp\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349922    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flcc2\" (UniqueName: \"kubernetes.io/projected/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-kube-api-access-flcc2\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349959    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c546cea-b474-44cf-84a5-b89784bda1d6-image-registry-private-configuration\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349988    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-default-certificate\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.349991    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/354d7018-887a-497f-b74f-195c8c88e184-snapshots\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350074    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e49068d1-9ae4-47b3-9a07-3d695a0a2402-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350078    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d0de4046-bfed-4695-af0e-1ba806e6b031-signing-key\") pod \"service-ca-5fffc97bdf-qlt9s\" (UID: \"d0de4046-bfed-4695-af0e-1ba806e6b031\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlt9s"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350086    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-tmp\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350119    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-run-systemd\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350129    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/354d7018-887a-497f-b74f-195c8c88e184-snapshots\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350155    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-run-openvswitch\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350169    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c546cea-b474-44cf-84a5-b89784bda1d6-image-registry-private-configuration\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350177    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350214    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9260d15b-9772-4cb4-8588-8e82f89f39b8-serviceca\") pod \"node-ca-qg2kq\" (UID: \"9260d15b-9772-4cb4-8588-8e82f89f39b8\") " pod="openshift-image-registry/node-ca-qg2kq"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350236    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-certificates\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350252    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/62c3ca06-b2a4-49d7-84db-1390277be2c8-archives-path\") pod \"periodic-gathering-gxg2j-p5824\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") " pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350273    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f4425c17-5cc6-4afc-922c-d74694805196-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350297    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d0de4046-bfed-4695-af0e-1ba806e6b031-signing-cabundle\") pod \"service-ca-5fffc97bdf-qlt9s\" (UID: \"d0de4046-bfed-4695-af0e-1ba806e6b031\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlt9s"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350320    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c546cea-b474-44cf-84a5-b89784bda1d6-ca-trust-extracted\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350335    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/62c3ca06-b2a4-49d7-84db-1390277be2c8-archives-path\") pod \"periodic-gathering-gxg2j-p5824\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") " pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:51:14.350328 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350345    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-etc-kubernetes\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350372    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e49068d1-9ae4-47b3-9a07-3d695a0a2402-cnibin\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350390    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-certificates\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350408    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c546cea-b474-44cf-84a5-b89784bda1d6-ca-trust-extracted\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350426    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9260d15b-9772-4cb4-8588-8e82f89f39b8-serviceca\") pod \"node-ca-qg2kq\" (UID: \"9260d15b-9772-4cb4-8588-8e82f89f39b8\") " pod="openshift-image-registry/node-ca-qg2kq"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350415    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z56p\" (UniqueName: \"kubernetes.io/projected/be449663-2797-487f-8559-0a96c563ae3a-kube-api-access-5z56p\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350468    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc9gg\" (UniqueName: \"kubernetes.io/projected/baf5281f-5938-4417-b22f-2644034da7ef-kube-api-access-mc9gg\") pod \"service-ca-operator-544d749c8c-nw7sr\" (UID: \"baf5281f-5938-4417-b22f-2644034da7ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350485    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85997\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-kube-api-access-85997\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350510    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-run-netns\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350527    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f4425c17-5cc6-4afc-922c-d74694805196-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350618    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-log-socket\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350638    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvkdd\" (UniqueName: \"kubernetes.io/projected/665f9d02-067e-4506-9793-cc7f17eb95bb-kube-api-access-cvkdd\") pod \"network-check-source-6f78d59f99-z28hp\" (UID: \"665f9d02-067e-4506-9793-cc7f17eb95bb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z28hp"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350660    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f0923bb-db25-4266-ad8d-2e1cfb7f602b-konnectivity-ca\") pod \"konnectivity-agent-vvzws\" (UID: \"1f0923bb-db25-4266-ad8d-2e1cfb7f602b\") " pod="kube-system/konnectivity-agent-vvzws"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350684    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be449663-2797-487f-8559-0a96c563ae3a-crio-socket\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350791    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5lg2\" (UniqueName: \"kubernetes.io/projected/2555270f-a2ea-4d11-8dff-d130f9786e3d-kube-api-access-q5lg2\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350810    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62c3ca06-b2a4-49d7-84db-1390277be2c8-service-ca-bundle\") pod \"periodic-gathering-gxg2j-p5824\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") " pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350828    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg6dt\" (UniqueName: \"kubernetes.io/projected/354d7018-887a-497f-b74f-195c8c88e184-kube-api-access-fg6dt\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350855    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-registration-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350885    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/108d9224-ccd3-4925-a65b-0d59e938ff72-image-registry-private-configuration\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350912    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2d5x\" (UniqueName: \"kubernetes.io/projected/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-kube-api-access-f2d5x\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350953    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f0923bb-db25-4266-ad8d-2e1cfb7f602b-konnectivity-ca\") pod \"konnectivity-agent-vvzws\" (UID: \"1f0923bb-db25-4266-ad8d-2e1cfb7f602b\") " pod="kube-system/konnectivity-agent-vvzws"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350978    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btsvv\" (UniqueName: \"kubernetes.io/projected/3a6655a6-9079-4927-8e6c-09d075de438c-kube-api-access-btsvv\") pod \"kas-connection-checker-685895c7c9-rcv8q\" (UID: \"3a6655a6-9079-4927-8e6c-09d075de438c\") " pod="kube-system/kas-connection-checker-685895c7c9-rcv8q"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.350978    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62c3ca06-b2a4-49d7-84db-1390277be2c8-service-ca-bundle\") pod \"periodic-gathering-gxg2j-p5824\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") " pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351087    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhg5z\" (UniqueName: \"kubernetes.io/projected/9260d15b-9772-4cb4-8588-8e82f89f39b8-kube-api-access-hhg5z\") pod \"node-ca-qg2kq\" (UID: \"9260d15b-9772-4cb4-8588-8e82f89f39b8\") " pod="openshift-image-registry/node-ca-qg2kq"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351106    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/354d7018-887a-497f-b74f-195c8c88e184-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351123    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-cni-dir\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351132    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/108d9224-ccd3-4925-a65b-0d59e938ff72-image-registry-private-configuration\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351138    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-daemon-config\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351197    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcpqh\" (UniqueName: \"kubernetes.io/projected/e49068d1-9ae4-47b3-9a07-3d695a0a2402-kube-api-access-jcpqh\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351221    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t8q9\" (UniqueName: \"kubernetes.io/projected/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-kube-api-access-6t8q9\") pod \"console-operator-7cd56b8dbc-rjmbd\" (UID: \"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351237    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-stats-auth\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351251    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f0923bb-db25-4266-ad8d-2e1cfb7f602b-agent-certs\") pod \"konnectivity-agent-vvzws\" (UID: \"1f0923bb-db25-4266-ad8d-2e1cfb7f602b\") " pod="kube-system/konnectivity-agent-vvzws"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351272    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-sys-fs\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351327    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-daemon-config\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351374    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-var-lib-openvswitch\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351389    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/354d7018-887a-497f-b74f-195c8c88e184-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351401    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-run-ovn\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351432    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-ovnkube-config\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351537    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e49068d1-9ae4-47b3-9a07-3d695a0a2402-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351572    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-55bnn\" (UID: \"0110b28b-ec04-4f65-bc3d-2e82f88b30d7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351587    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-stats-auth\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351596    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-ovnkube-config\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351610    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkrns\" (UniqueName: \"kubernetes.io/projected/d0de4046-bfed-4695-af0e-1ba806e6b031-kube-api-access-dkrns\") pod \"service-ca-5fffc97bdf-qlt9s\" (UID: \"d0de4046-bfed-4695-af0e-1ba806e6b031\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlt9s"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351653    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f0923bb-db25-4266-ad8d-2e1cfb7f602b-agent-certs\") pod \"konnectivity-agent-vvzws\" (UID: \"1f0923bb-db25-4266-ad8d-2e1cfb7f602b\") " pod="kube-system/konnectivity-agent-vvzws"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351721    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkw2h\" (UniqueName: \"kubernetes.io/projected/ef90f36a-279e-4bfa-a97d-70132a266670-kube-api-access-kkw2h\") pod \"cluster-samples-operator-6ddb6c8b94-qtpnh\" (UID: \"ef90f36a-279e-4bfa-a97d-70132a266670\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351769    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzfk9\" (UniqueName: \"kubernetes.io/projected/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-kube-api-access-rzfk9\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351786    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-55bnn\" (UID: \"0110b28b-ec04-4f65-bc3d-2e82f88b30d7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351793    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-systemd\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351829    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qtpnh\" (UID: \"ef90f36a-279e-4bfa-a97d-70132a266670\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351874    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-systemd-units\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351926    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-ovn-node-metrics-cert\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351945    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c6wq\" (UniqueName: \"kubernetes.io/projected/b25bd7be-290a-499a-a81d-c005793eace2-kube-api-access-8c6wq\") pod \"ingress-canary-7t99k\" (UID: \"b25bd7be-290a-499a-a81d-c005793eace2\") " pod="openshift-ingress-canary/ingress-canary-7t99k"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351963    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2555270f-a2ea-4d11-8dff-d130f9786e3d-config-volume\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.351985    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/108d9224-ccd3-4925-a65b-0d59e938ff72-ca-trust-extracted\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.352007    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-bound-sa-token\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.352025    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-socket-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.352100    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/108d9224-ccd3-4925-a65b-0d59e938ff72-ca-trust-extracted\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.352106    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-ovn-node-metrics-cert\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.352166 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.352172    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2555270f-a2ea-4d11-8dff-d130f9786e3d-config-volume\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:14.357797 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.357782    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 01:51:14.357860 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.357845    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 01:51:14.357899 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.357886    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 01:51:14.357941 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.357901    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 01:51:14.359249 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.359235    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b57cc991-d758-435f-9238-0dac97f10a93-iptables-alerter-script\") pod \"iptables-alerter-tp5hd\" (UID: \"b57cc991-d758-435f-9238-0dac97f10a93\") " pod="openshift-network-operator/iptables-alerter-tp5hd"
Nov 03 01:51:14.361199 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.361165    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d0de4046-bfed-4695-af0e-1ba806e6b031-signing-cabundle\") pod \"service-ca-5fffc97bdf-qlt9s\" (UID: \"d0de4046-bfed-4695-af0e-1ba806e6b031\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlt9s"
Nov 03 01:51:14.398984 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.398958    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ln9h\" (UniqueName: \"kubernetes.io/projected/38de413a-873a-4288-85f5-c453581a53d6-kube-api-access-9ln9h\") pod \"kas-connection-checker-685895c7c9-l87nq\" (UID: \"38de413a-873a-4288-85f5-c453581a53d6\") " pod="kube-system/kas-connection-checker-685895c7c9-l87nq"
Nov 03 01:51:14.399434 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.399411    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z46t7\" (UniqueName: \"kubernetes.io/projected/62c3ca06-b2a4-49d7-84db-1390277be2c8-kube-api-access-z46t7\") pod \"periodic-gathering-gxg2j-p5824\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") " pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:51:14.399522 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.399483    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg5wd\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-kube-api-access-cg5wd\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:14.399522 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.399488    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h48h\" (UniqueName: \"kubernetes.io/projected/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-kube-api-access-5h48h\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.399867 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.399850    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxvzg\" (UniqueName: \"kubernetes.io/projected/48604fe3-9728-461e-ae37-8208b424e149-kube-api-access-zxvzg\") pod \"volume-data-source-validator-58c9f5f46d-f5dm4\" (UID: \"48604fe3-9728-461e-ae37-8208b424e149\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-f5dm4"
Nov 03 01:51:14.401731 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.401711    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5jp6\" (UniqueName: \"kubernetes.io/projected/62150808-f089-4723-aac5-d17d3a803abe-kube-api-access-g5jp6\") pod \"migrator-66df5d496b-t9rqp\" (UID: \"62150808-f089-4723-aac5-d17d3a803abe\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t9rqp"
Nov 03 01:51:14.401871 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.401852    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl96f\" (UniqueName: \"kubernetes.io/projected/b57cc991-d758-435f-9238-0dac97f10a93-kube-api-access-sl96f\") pod \"iptables-alerter-tp5hd\" (UID: \"b57cc991-d758-435f-9238-0dac97f10a93\") " pod="openshift-network-operator/iptables-alerter-tp5hd"
Nov 03 01:51:14.402455 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.402326    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-bound-sa-token\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:14.402455 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.402353    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n62b\" (UniqueName: \"kubernetes.io/projected/f4425c17-5cc6-4afc-922c-d74694805196-kube-api-access-8n62b\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:14.402455 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.402374    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqqkf\" (UniqueName: \"kubernetes.io/projected/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-kube-api-access-qqqkf\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:51:14.402455 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.402384    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltkx2\" (UniqueName: \"kubernetes.io/projected/c7a013c2-18a4-4372-adcb-8dfafa673f9c-kube-api-access-ltkx2\") pod \"network-metrics-daemon-xv8zr\" (UID: \"c7a013c2-18a4-4372-adcb-8dfafa673f9c\") " pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:51:14.402455 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.402378    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xbq7\" (UniqueName: \"kubernetes.io/projected/e11ea02d-ac52-4318-9fdd-95c0bb171aed-kube-api-access-2xbq7\") pod \"kas-connection-checker-685895c7c9-jqkfr\" (UID: \"e11ea02d-ac52-4318-9fdd-95c0bb171aed\") " pod="kube-system/kas-connection-checker-685895c7c9-jqkfr"
Nov 03 01:51:14.402455 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.402428    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqsmb\" (UniqueName: \"kubernetes.io/projected/15face97-689e-4fef-86c0-17a2a85829db-kube-api-access-zqsmb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t59nn\" (UID: \"15face97-689e-4fef-86c0-17a2a85829db\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn"
Nov 03 01:51:14.402455 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.402441    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq5lq\" (UniqueName: \"kubernetes.io/projected/42a56b4b-283d-4e63-bb6a-59c5d5df0c2c-kube-api-access-hq5lq\") pod \"network-check-target-mdr8p\" (UID: \"42a56b4b-283d-4e63-bb6a-59c5d5df0c2c\") " pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:51:14.402455 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.402443    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6lhl\" (UniqueName: \"kubernetes.io/projected/8684a429-234f-4537-917e-5b49509bbdb5-kube-api-access-h6lhl\") pod \"node-resolver-p8kgq\" (UID: \"8684a429-234f-4537-917e-5b49509bbdb5\") " pod="openshift-dns/node-resolver-p8kgq"
Nov 03 01:51:14.402777 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.402756    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flcc2\" (UniqueName: \"kubernetes.io/projected/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-kube-api-access-flcc2\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.403396 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.403379    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z56p\" (UniqueName: \"kubernetes.io/projected/be449663-2797-487f-8559-0a96c563ae3a-kube-api-access-5z56p\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:14.403511 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.403496    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f76x5\" (UniqueName: \"kubernetes.io/projected/cb8ca898-6f5f-4727-9d6d-d6aed65adfae-kube-api-access-f76x5\") pod \"migrator-66df5d496b-4wq24\" (UID: \"cb8ca898-6f5f-4727-9d6d-d6aed65adfae\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4wq24"
Nov 03 01:51:14.404500 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.404484    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-bound-sa-token\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:14.405541 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.405526    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc9gg\" (UniqueName: \"kubernetes.io/projected/baf5281f-5938-4417-b22f-2644034da7ef-kube-api-access-mc9gg\") pod \"service-ca-operator-544d749c8c-nw7sr\" (UID: \"baf5281f-5938-4417-b22f-2644034da7ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr"
Nov 03 01:51:14.405727 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.405712    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhg5z\" (UniqueName: \"kubernetes.io/projected/9260d15b-9772-4cb4-8588-8e82f89f39b8-kube-api-access-hhg5z\") pod \"node-ca-qg2kq\" (UID: \"9260d15b-9772-4cb4-8588-8e82f89f39b8\") " pod="openshift-image-registry/node-ca-qg2kq"
Nov 03 01:51:14.406089 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.406068    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg6dt\" (UniqueName: \"kubernetes.io/projected/354d7018-887a-497f-b74f-195c8c88e184-kube-api-access-fg6dt\") pod \"insights-operator-7c4986d449-gqz7v\" (UID: \"354d7018-887a-497f-b74f-195c8c88e184\") " pod="openshift-insights/insights-operator-7c4986d449-gqz7v"
Nov 03 01:51:14.406806 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.406276    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85997\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-kube-api-access-85997\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:14.406806 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.406643    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btsvv\" (UniqueName: \"kubernetes.io/projected/3a6655a6-9079-4927-8e6c-09d075de438c-kube-api-access-btsvv\") pod \"kas-connection-checker-685895c7c9-rcv8q\" (UID: \"3a6655a6-9079-4927-8e6c-09d075de438c\") " pod="kube-system/kas-connection-checker-685895c7c9-rcv8q"
Nov 03 01:51:14.406806 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.406679    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c6wq\" (UniqueName: \"kubernetes.io/projected/b25bd7be-290a-499a-a81d-c005793eace2-kube-api-access-8c6wq\") pod \"ingress-canary-7t99k\" (UID: \"b25bd7be-290a-499a-a81d-c005793eace2\") " pod="openshift-ingress-canary/ingress-canary-7t99k"
Nov 03 01:51:14.406806 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.406681    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzfk9\" (UniqueName: \"kubernetes.io/projected/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-kube-api-access-rzfk9\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:14.406806 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.406755    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkw2h\" (UniqueName: \"kubernetes.io/projected/ef90f36a-279e-4bfa-a97d-70132a266670-kube-api-access-kkw2h\") pod \"cluster-samples-operator-6ddb6c8b94-qtpnh\" (UID: \"ef90f36a-279e-4bfa-a97d-70132a266670\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:14.406806 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.406776    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2d5x\" (UniqueName: \"kubernetes.io/projected/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-kube-api-access-f2d5x\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.407070 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.406856    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvkdd\" (UniqueName: \"kubernetes.io/projected/665f9d02-067e-4506-9793-cc7f17eb95bb-kube-api-access-cvkdd\") pod \"network-check-source-6f78d59f99-z28hp\" (UID: \"665f9d02-067e-4506-9793-cc7f17eb95bb\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-z28hp"
Nov 03 01:51:14.407070 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.406898    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5lg2\" (UniqueName: \"kubernetes.io/projected/2555270f-a2ea-4d11-8dff-d130f9786e3d-kube-api-access-q5lg2\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:14.407070 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.406941    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t8q9\" (UniqueName: \"kubernetes.io/projected/a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd-kube-api-access-6t8q9\") pod \"console-operator-7cd56b8dbc-rjmbd\" (UID: \"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:14.407236 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.407216    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcpqh\" (UniqueName: \"kubernetes.io/projected/e49068d1-9ae4-47b3-9a07-3d695a0a2402-kube-api-access-jcpqh\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:51:14.407516 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.407501    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkrns\" (UniqueName: \"kubernetes.io/projected/d0de4046-bfed-4695-af0e-1ba806e6b031-kube-api-access-dkrns\") pod \"service-ca-5fffc97bdf-qlt9s\" (UID: \"d0de4046-bfed-4695-af0e-1ba806e6b031\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlt9s"
Nov 03 01:51:14.452931 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.452909    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-sysconfig\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.453113 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.452935    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-run-k8s-cni-cncf-io\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.453113 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.452950    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:51:14.453113 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.452968    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-hostroot\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.453113 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.452984    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8684a429-234f-4537-917e-5b49509bbdb5-hosts-file\") pod \"node-resolver-p8kgq\" (UID: \"8684a429-234f-4537-917e-5b49509bbdb5\") " pod="openshift-dns/node-resolver-p8kgq"
Nov 03 01:51:14.453113 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453017    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-run-k8s-cni-cncf-io\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.453113 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453037    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:51:14.453113 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453046    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-hostroot\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.453113 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453024    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-socket-dir-parent\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.453113 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453061    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-sysconfig\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.453113 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453069    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-socket-dir-parent\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.453113 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453099    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs\") pod \"network-metrics-daemon-xv8zr\" (UID: \"c7a013c2-18a4-4372-adcb-8dfafa673f9c\") " pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:51:14.453113 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453097    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8684a429-234f-4537-917e-5b49509bbdb5-hosts-file\") pod \"node-resolver-p8kgq\" (UID: \"8684a429-234f-4537-917e-5b49509bbdb5\") " pod="openshift-dns/node-resolver-p8kgq"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453155    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.453173    5005 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453200    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-run-netns\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453174    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-run-netns\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.453244    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs podName:c7a013c2-18a4-4372-adcb-8dfafa673f9c nodeName:}" failed. No retries permitted until 2026-11-03 01:51:14.953226617 +0000 UTC m=+1.967467979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs") pod "network-metrics-daemon-xv8zr" (UID: "c7a013c2-18a4-4372-adcb-8dfafa673f9c") : secret "metrics-daemon-secret" not found
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453263    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.453273    5005 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453283    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-run-systemd\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453303    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-run-openvswitch\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.453330    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls podName:f4425c17-5cc6-4afc-922c-d74694805196 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:14.953317989 +0000 UTC m=+1.967559361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-g8wz8" (UID: "f4425c17-5cc6-4afc-922c-d74694805196") : secret "cluster-monitoring-operator-tls" not found
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453334    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-run-openvswitch\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453343    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453365    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-run-systemd\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.453371    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle podName:7a9e57fd-a6e6-4ddb-aa08-ac1390403544 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:14.953358639 +0000 UTC m=+1.967600007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle") pod "router-default-7875dd7849-ptjkr" (UID: "7a9e57fd-a6e6-4ddb-aa08-ac1390403544") : configmap references non-existent config key: service-ca.crt
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.453389    5005 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453391    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-etc-kubernetes\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453417    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-etc-kubernetes\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453407    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e49068d1-9ae4-47b3-9a07-3d695a0a2402-cnibin\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.453435    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs podName:7a9e57fd-a6e6-4ddb-aa08-ac1390403544 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:14.953427202 +0000 UTC m=+1.967668545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs") pod "router-default-7875dd7849-ptjkr" (UID: "7a9e57fd-a6e6-4ddb-aa08-ac1390403544") : secret "router-metrics-certs-default" not found
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453447    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e49068d1-9ae4-47b3-9a07-3d695a0a2402-cnibin\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453460    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-run-netns\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453483    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-log-socket\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453495    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-run-netns\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453501    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be449663-2797-487f-8559-0a96c563ae3a-crio-socket\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453520    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-registration-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453537    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-cni-dir\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453554    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-sys-fs\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453535    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-log-socket\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453569    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-var-lib-openvswitch\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453574    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be449663-2797-487f-8559-0a96c563ae3a-crio-socket\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453586    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-registration-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453598    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-run-ovn\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453600    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-var-lib-openvswitch\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453602    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-cni-dir\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.453615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453631    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e49068d1-9ae4-47b3-9a07-3d695a0a2402-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453636    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-run-ovn\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453636    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-sys-fs\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453676    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-systemd\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453678    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e49068d1-9ae4-47b3-9a07-3d695a0a2402-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453734    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qtpnh\" (UID: \"ef90f36a-279e-4bfa-a97d-70132a266670\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453746    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-systemd\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453767    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-systemd-units\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.453814    5005 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453820    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-socket-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453835    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-systemd-units\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453845    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-kubelet\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.453868    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls podName:ef90f36a-279e-4bfa-a97d-70132a266670 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:14.953854623 +0000 UTC m=+1.968095967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qtpnh" (UID: "ef90f36a-279e-4bfa-a97d-70132a266670") : secret "samples-operator-tls" not found
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453884    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-socket-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453885    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-kubelet\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453886    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-cni-bin\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453917    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be449663-2797-487f-8559-0a96c563ae3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453934    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-run\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453918    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-cni-bin\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453950    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-host\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453983    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b57cc991-d758-435f-9238-0dac97f10a93-host-slash\") pod \"iptables-alerter-tp5hd\" (UID: \"b57cc991-d758-435f-9238-0dac97f10a93\") " pod="openshift-network-operator/iptables-alerter-tp5hd"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453989    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-host\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.453993    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-run\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454024    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b57cc991-d758-435f-9238-0dac97f10a93-host-slash\") pod \"iptables-alerter-tp5hd\" (UID: \"b57cc991-d758-435f-9238-0dac97f10a93\") " pod="openshift-network-operator/iptables-alerter-tp5hd"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454023    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-lib-modules\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454045    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79fe02fa-99ce-4222-b569-6b323df47ab4-kubelet-config\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454059    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e49068d1-9ae4-47b3-9a07-3d695a0a2402-system-cni-dir\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454079    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79fe02fa-99ce-4222-b569-6b323df47ab4-dbus\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454097    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454097    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79fe02fa-99ce-4222-b569-6b323df47ab4-kubelet-config\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454111    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-device-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454123    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-lib-modules\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454128    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454151    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79fe02fa-99ce-4222-b569-6b323df47ab4-dbus\") pod \"global-pull-secret-syncer-mxg52\" (UID: \"79fe02fa-99ce-4222-b569-6b323df47ab4\") " pod="kube-system/global-pull-secret-syncer-mxg52"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454156    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454165    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454177    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e49068d1-9ae4-47b3-9a07-3d695a0a2402-system-cni-dir\") pod \"multus-additional-cni-plugins-d7qsp\" (UID: \"e49068d1-9ae4-47b3-9a07-3d695a0a2402\") " pod="openshift-multus/multus-additional-cni-plugins-d7qsp"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454195    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f30f499a-7d25-4ea7-aa1e-357ccbf0485f-device-dir\") pod \"aws-ebs-csi-driver-node-qh2vs\" (UID: \"f30f499a-7d25-4ea7-aa1e-357ccbf0485f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qh2vs"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.454202    5005 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.454216    5005 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55fd5cff67-bb76l: secret "image-registry-tls" not found
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454220    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.454229    5005 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.454241    5005 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bcfc5667d-w8v7m: secret "image-registry-tls" not found
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.454257    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls podName:9c546cea-b474-44cf-84a5-b89784bda1d6 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:14.95424534 +0000 UTC m=+1.968486687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls") pod "image-registry-55fd5cff67-bb76l" (UID: "9c546cea-b474-44cf-84a5-b89784bda1d6") : secret "image-registry-tls" not found
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454191    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.454288    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls podName:108d9224-ccd3-4925-a65b-0d59e938ff72 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:14.954275475 +0000 UTC m=+1.968516822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls") pod "image-registry-6bcfc5667d-w8v7m" (UID: "108d9224-ccd3-4925-a65b-0d59e938ff72") : secret "image-registry-tls" not found
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454305    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9260d15b-9772-4cb4-8588-8e82f89f39b8-host\") pod \"node-ca-qg2kq\" (UID: \"9260d15b-9772-4cb4-8588-8e82f89f39b8\") " pod="openshift-image-registry/node-ca-qg2kq"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454330    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-system-cni-dir\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454355    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-cnibin\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454366    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9260d15b-9772-4cb4-8588-8e82f89f39b8-host\") pod \"node-ca-qg2kq\" (UID: \"9260d15b-9772-4cb4-8588-8e82f89f39b8\") " pod="openshift-image-registry/node-ca-qg2kq"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454379    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-slash\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454384    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-system-cni-dir\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454404    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-etc-openvswitch\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454416    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-cnibin\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454429    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-node-log\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454432    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be449663-2797-487f-8559-0a96c563ae3a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454447    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-slash\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454454    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454482    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-sys\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.454499    5005 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454505    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-var-lib-kubelet\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.454538    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls podName:be449663-2797-487f-8559-0a96c563ae3a nodeName:}" failed. No retries permitted until 2026-11-03 01:51:14.954530238 +0000 UTC m=+1.968771581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9twrb" (UID: "be449663-2797-487f-8559-0a96c563ae3a") : secret "insights-runtime-extractor-tls" not found
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454538    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-sys\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454544    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-var-lib-kubelet\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454549    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-run-multus-certs\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454481    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-node-log\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454578    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-run-ovn-kubernetes\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454595    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-kubernetes\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454609    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-var-lib-kubelet\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454635    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-55bnn\" (UID: \"0110b28b-ec04-4f65-bc3d-2e82f88b30d7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454641    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-run-ovn-kubernetes\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454640    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-run-multus-certs\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454454    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-etc-openvswitch\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454655    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-kubernetes\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454663    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-var-lib-cni-multus\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454671    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-var-lib-kubelet\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454704    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-var-lib-cni-multus\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454713    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.454728    5005 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454733    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-sysctl-d\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454747    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-var-lib-cni-bin\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.454759    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert podName:0110b28b-ec04-4f65-bc3d-2e82f88b30d7 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:14.954750261 +0000 UTC m=+1.968991623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-55bnn" (UID: "0110b28b-ec04-4f65-bc3d-2e82f88b30d7") : secret "networking-console-plugin-cert" not found
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.454774    5005 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454778    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-cni-netd\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454795    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-modprobe-d\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.454803    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls podName:2555270f-a2ea-4d11-8dff-d130f9786e3d nodeName:}" failed. No retries permitted until 2026-11-03 01:51:14.954792821 +0000 UTC m=+1.969034165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls") pod "dns-default-w4h82" (UID: "2555270f-a2ea-4d11-8dff-d130f9786e3d") : secret "dns-default-metrics-tls" not found
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454772    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-host-var-lib-cni-bin\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454826    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-sysctl-conf\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454848    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-conf-dir\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454859    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/54ba28bb-4bab-4c0d-aa00-8b3bce675f23-host-cni-netd\") pod \"ovnkube-node-jdk59\" (UID: \"54ba28bb-4bab-4c0d-aa00-8b3bce675f23\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454863    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-sysctl-d\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454866    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-modprobe-d\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454900    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a7a1afb-11cc-4aa4-bae6-68ce495ebca8-multus-conf-dir\") pod \"multus-9kq9k\" (UID: \"1a7a1afb-11cc-4aa4-bae6-68ce495ebca8\") " pod="openshift-multus/multus-9kq9k"
Nov 03 01:51:14.455310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.454962    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0c1ca62b-2da2-47a9-8e2a-34f2dc75a431-etc-sysctl-conf\") pod \"tuned-25wbg\" (UID: \"0c1ca62b-2da2-47a9-8e2a-34f2dc75a431\") " pod="openshift-cluster-node-tuning-operator/tuned-25wbg"
Nov 03 01:51:14.511676 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.511657    5005 scope.go:122] "RemoveContainer" containerID="8df6474c221450b3bf6a0de2c5673ab2c142dbf0053634b6d05d2b7c199801e8"
Nov 03 01:51:14.527290 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.527272    5005 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:51:14.527853 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.527832    5005 scope.go:122] "RemoveContainer" containerID="94e591a01809801bcdd677e56da974da92c97951b24c49a9c4e7c65eeac3820a"
Nov 03 01:51:14.959032 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.958941    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:14.959032 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.958986    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:14.959032 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.959031    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qtpnh\" (UID: \"ef90f36a-279e-4bfa-a97d-70132a266670\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:14.959284 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.959062    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:14.959284 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.959079    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:14.959284 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.959100    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:14.959284 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.959129    5005 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 01:51:14.959284 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.959167    5005 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 01:51:14.959284 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.959177    5005 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bcfc5667d-w8v7m: secret "image-registry-tls" not found
Nov 03 01:51:14.959284 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.959179    5005 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 01:51:14.959284 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.959195    5005 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55fd5cff67-bb76l: secret "image-registry-tls" not found
Nov 03 01:51:14.959284 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.959201    5005 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 01:51:14.959284 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.959133    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle podName:7a9e57fd-a6e6-4ddb-aa08-ac1390403544 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:15.959110849 +0000 UTC m=+2.973352193 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle") pod "router-default-7875dd7849-ptjkr" (UID: "7a9e57fd-a6e6-4ddb-aa08-ac1390403544") : configmap references non-existent config key: service-ca.crt
Nov 03 01:51:14.959284 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.959241    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs podName:7a9e57fd-a6e6-4ddb-aa08-ac1390403544 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:15.959229224 +0000 UTC m=+2.973470573 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs") pod "router-default-7875dd7849-ptjkr" (UID: "7a9e57fd-a6e6-4ddb-aa08-ac1390403544") : secret "router-metrics-certs-default" not found
Nov 03 01:51:14.959284 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.959260    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls podName:108d9224-ccd3-4925-a65b-0d59e938ff72 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:15.959250717 +0000 UTC m=+2.973492062 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls") pod "image-registry-6bcfc5667d-w8v7m" (UID: "108d9224-ccd3-4925-a65b-0d59e938ff72") : secret "image-registry-tls" not found
Nov 03 01:51:14.959284 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.959267    5005 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 01:51:14.959813 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.959274    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls podName:9c546cea-b474-44cf-84a5-b89784bda1d6 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:15.959269113 +0000 UTC m=+2.973510457 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls") pod "image-registry-55fd5cff67-bb76l" (UID: "9c546cea-b474-44cf-84a5-b89784bda1d6") : secret "image-registry-tls" not found
Nov 03 01:51:14.959813 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.959330    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls podName:ef90f36a-279e-4bfa-a97d-70132a266670 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:15.959318764 +0000 UTC m=+2.973560111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qtpnh" (UID: "ef90f36a-279e-4bfa-a97d-70132a266670") : secret "samples-operator-tls" not found
Nov 03 01:51:14.959813 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.959343    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls podName:be449663-2797-487f-8559-0a96c563ae3a nodeName:}" failed. No retries permitted until 2026-11-03 01:51:15.959336904 +0000 UTC m=+2.973578248 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9twrb" (UID: "be449663-2797-487f-8559-0a96c563ae3a") : secret "insights-runtime-extractor-tls" not found
Nov 03 01:51:14.959813 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.959377    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-55bnn\" (UID: \"0110b28b-ec04-4f65-bc3d-2e82f88b30d7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn"
Nov 03 01:51:14.959813 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.959400    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:14.959813 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.959431    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs\") pod \"network-metrics-daemon-xv8zr\" (UID: \"c7a013c2-18a4-4372-adcb-8dfafa673f9c\") " pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:51:14.959813 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:14.959448    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:14.959813 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.959472    5005 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 01:51:14.959813 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.959495    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert podName:0110b28b-ec04-4f65-bc3d-2e82f88b30d7 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:15.959489339 +0000 UTC m=+2.973730683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-55bnn" (UID: "0110b28b-ec04-4f65-bc3d-2e82f88b30d7") : secret "networking-console-plugin-cert" not found
Nov 03 01:51:14.959813 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.959520    5005 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 01:51:14.959813 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.959548    5005 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 01:51:14.959813 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.959584    5005 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 01:51:14.959813 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.959552    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls podName:f4425c17-5cc6-4afc-922c-d74694805196 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:15.959541795 +0000 UTC m=+2.973783144 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-g8wz8" (UID: "f4425c17-5cc6-4afc-922c-d74694805196") : secret "cluster-monitoring-operator-tls" not found
Nov 03 01:51:14.959813 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.959619    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls podName:2555270f-a2ea-4d11-8dff-d130f9786e3d nodeName:}" failed. No retries permitted until 2026-11-03 01:51:15.9596111 +0000 UTC m=+2.973852444 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls") pod "dns-default-w4h82" (UID: "2555270f-a2ea-4d11-8dff-d130f9786e3d") : secret "dns-default-metrics-tls" not found
Nov 03 01:51:14.959813 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:14.959628    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs podName:c7a013c2-18a4-4372-adcb-8dfafa673f9c nodeName:}" failed. No retries permitted until 2026-11-03 01:51:15.959623268 +0000 UTC m=+2.973864612 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs") pod "network-metrics-daemon-xv8zr" (UID: "c7a013c2-18a4-4372-adcb-8dfafa673f9c") : secret "metrics-daemon-secret" not found
Nov 03 01:51:15.066649 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:15.066627    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p8kgq_8684a429-234f-4537-917e-5b49509bbdb5/dns-node-resolver/0.log"
Nov 03 01:51:15.163042 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:15.163012    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-qlt9s" event={"ID":"d0de4046-bfed-4695-af0e-1ba806e6b031","Type":"ContainerStarted","Data":"6c948c82aeb71d45ef5ff33ec6d82dce83d38cb5267e2d0e2d3d505e2ccbb36a"}
Nov 03 01:51:15.164709 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:15.164681    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 01:51:15.165084 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:15.165071    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/1.log"
Nov 03 01:51:15.165144 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:15.165101    5005 generic.go:356] "Generic (PLEG): container finished" podID="a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd" containerID="621ffc29ff310c22fafdd982e60f04f27b6d2cbafd590f6936927afc32e6259c" exitCode=255
Nov 03 01:51:15.165208 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:15.165185    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd" event={"ID":"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd","Type":"ContainerDied","Data":"621ffc29ff310c22fafdd982e60f04f27b6d2cbafd590f6936927afc32e6259c"}
Nov 03 01:51:15.165263 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:15.165246    5005 scope.go:122] "RemoveContainer" containerID="8df6474c221450b3bf6a0de2c5673ab2c142dbf0053634b6d05d2b7c199801e8"
Nov 03 01:51:15.165640 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:15.165616    5005 scope.go:122] "RemoveContainer" containerID="621ffc29ff310c22fafdd982e60f04f27b6d2cbafd590f6936927afc32e6259c"
Nov 03 01:51:15.165846 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:15.165820    5005 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rjmbd_openshift-console-operator(a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd" podUID="a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd"
Nov 03 01:51:15.970145 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:15.970106    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qtpnh\" (UID: \"ef90f36a-279e-4bfa-a97d-70132a266670\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:15.970349 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:15.970168    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:15.970349 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:15.970190    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:15.970349 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:15.970212    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:15.970349 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:15.970243    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-55bnn\" (UID: \"0110b28b-ec04-4f65-bc3d-2e82f88b30d7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn"
Nov 03 01:51:15.970349 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:15.970262    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:15.970349 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:15.970290    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs\") pod \"network-metrics-daemon-xv8zr\" (UID: \"c7a013c2-18a4-4372-adcb-8dfafa673f9c\") " pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:51:15.970349 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:15.970308    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:15.970349 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:15.970328    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:15.970349 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:15.970345    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:15.970815 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:15.970459    5005 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 01:51:15.970815 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:15.970502    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs podName:7a9e57fd-a6e6-4ddb-aa08-ac1390403544 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:17.970490911 +0000 UTC m=+4.984732268 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs") pod "router-default-7875dd7849-ptjkr" (UID: "7a9e57fd-a6e6-4ddb-aa08-ac1390403544") : secret "router-metrics-certs-default" not found
Nov 03 01:51:15.970815 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:15.970555    5005 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 01:51:15.970815 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:15.970576    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls podName:ef90f36a-279e-4bfa-a97d-70132a266670 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:17.97056941 +0000 UTC m=+4.984810767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-qtpnh" (UID: "ef90f36a-279e-4bfa-a97d-70132a266670") : secret "samples-operator-tls" not found
Nov 03 01:51:15.970815 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:15.970617    5005 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 01:51:15.970815 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:15.970625    5005 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55fd5cff67-bb76l: secret "image-registry-tls" not found
Nov 03 01:51:15.970815 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:15.970644    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls podName:9c546cea-b474-44cf-84a5-b89784bda1d6 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:17.970638735 +0000 UTC m=+4.984880092 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls") pod "image-registry-55fd5cff67-bb76l" (UID: "9c546cea-b474-44cf-84a5-b89784bda1d6") : secret "image-registry-tls" not found
Nov 03 01:51:15.970815 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:15.970677    5005 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 01:51:15.970815 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:15.970684    5005 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bcfc5667d-w8v7m: secret "image-registry-tls" not found
Nov 03 01:51:15.970815 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:15.970728    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls podName:108d9224-ccd3-4925-a65b-0d59e938ff72 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:17.970722498 +0000 UTC m=+4.984963855 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls") pod "image-registry-6bcfc5667d-w8v7m" (UID: "108d9224-ccd3-4925-a65b-0d59e938ff72") : secret "image-registry-tls" not found
Nov 03 01:51:15.970815 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:15.970764    5005 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 01:51:15.970815 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:15.970785    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls podName:be449663-2797-487f-8559-0a96c563ae3a nodeName:}" failed. No retries permitted until 2026-11-03 01:51:17.970779792 +0000 UTC m=+4.985021135 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9twrb" (UID: "be449663-2797-487f-8559-0a96c563ae3a") : secret "insights-runtime-extractor-tls" not found
Nov 03 01:51:15.971390 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:15.970832    5005 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 01:51:15.971390 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:15.970851    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert podName:0110b28b-ec04-4f65-bc3d-2e82f88b30d7 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:17.970845677 +0000 UTC m=+4.985087034 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-55bnn" (UID: "0110b28b-ec04-4f65-bc3d-2e82f88b30d7") : secret "networking-console-plugin-cert" not found
Nov 03 01:51:15.971390 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:15.970891    5005 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 01:51:15.971390 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:15.970909    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls podName:2555270f-a2ea-4d11-8dff-d130f9786e3d nodeName:}" failed. No retries permitted until 2026-11-03 01:51:17.97090364 +0000 UTC m=+4.985144983 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls") pod "dns-default-w4h82" (UID: "2555270f-a2ea-4d11-8dff-d130f9786e3d") : secret "dns-default-metrics-tls" not found
Nov 03 01:51:15.971390 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:15.970941    5005 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 01:51:15.971390 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:15.970957    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs podName:c7a013c2-18a4-4372-adcb-8dfafa673f9c nodeName:}" failed. No retries permitted until 2026-11-03 01:51:17.970952006 +0000 UTC m=+4.985193350 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs") pod "network-metrics-daemon-xv8zr" (UID: "c7a013c2-18a4-4372-adcb-8dfafa673f9c") : secret "metrics-daemon-secret" not found
Nov 03 01:51:15.971390 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:15.970985    5005 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 01:51:15.971390 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:15.971002    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls podName:f4425c17-5cc6-4afc-922c-d74694805196 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:17.970997157 +0000 UTC m=+4.985238501 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-g8wz8" (UID: "f4425c17-5cc6-4afc-922c-d74694805196") : secret "cluster-monitoring-operator-tls" not found
Nov 03 01:51:15.971390 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:15.971041    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle podName:7a9e57fd-a6e6-4ddb-aa08-ac1390403544 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:17.971036052 +0000 UTC m=+4.985277396 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle") pod "router-default-7875dd7849-ptjkr" (UID: "7a9e57fd-a6e6-4ddb-aa08-ac1390403544") : configmap references non-existent config key: service-ca.crt
Nov 03 01:51:16.172578 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:16.172553    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 01:51:16.214844 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:16.214758    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qg2kq_9260d15b-9772-4cb4-8588-8e82f89f39b8/node-ca/0.log"
Nov 03 01:51:16.628640 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:16.628614    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7t99k_b25bd7be-290a-499a-a81d-c005793eace2/serve-healthcheck-canary/0.log"
Nov 03 01:51:17.177451 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:17.177416    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4wq24" event={"ID":"cb8ca898-6f5f-4727-9d6d-d6aed65adfae","Type":"ContainerStarted","Data":"597d26e4611c07faeafdfbc0ae6222aa8d5bbbb68a0dbaf8f5d59e8eb46100d2"}
Nov 03 01:51:17.178955 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:17.178934    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t9rqp" event={"ID":"62150808-f089-4723-aac5-d17d3a803abe","Type":"ContainerStarted","Data":"8dbf297f0f50cb442dea40558686bb13eb76d5c0b3a097e5eef48b54d99d611e"}
Nov 03 01:51:17.215196 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:17.215177    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-t59nn_15face97-689e-4fef-86c0-17a2a85829db/kube-storage-version-migrator-operator/0.log"
Nov 03 01:51:17.284223 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:17.284199    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:51:17.993729 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:17.993682    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs\") pod \"network-metrics-daemon-xv8zr\" (UID: \"c7a013c2-18a4-4372-adcb-8dfafa673f9c\") " pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:51:17.993729 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:17.993731    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:17.993982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:17.993760    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:17.993982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:17.993782    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:17.993982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:17.993816    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qtpnh\" (UID: \"ef90f36a-279e-4bfa-a97d-70132a266670\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:17.993982 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:17.993838    5005 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 01:51:17.993982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:17.993856    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:17.993982 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:17.993890    5005 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 01:51:17.993982 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:17.993909    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs podName:c7a013c2-18a4-4372-adcb-8dfafa673f9c nodeName:}" failed. No retries permitted until 2026-11-03 01:51:21.993889075 +0000 UTC m=+9.008130426 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs") pod "network-metrics-daemon-xv8zr" (UID: "c7a013c2-18a4-4372-adcb-8dfafa673f9c") : secret "metrics-daemon-secret" not found
Nov 03 01:51:17.993982 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:17.993931    5005 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 01:51:17.993982 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:17.993944    5005 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55fd5cff67-bb76l: secret "image-registry-tls" not found
Nov 03 01:51:17.993982 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:17.993985    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls podName:9c546cea-b474-44cf-84a5-b89784bda1d6 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:21.993971438 +0000 UTC m=+9.008212784 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls") pod "image-registry-55fd5cff67-bb76l" (UID: "9c546cea-b474-44cf-84a5-b89784bda1d6") : secret "image-registry-tls" not found
Nov 03 01:51:17.994417 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:17.993986    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:17.994417 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:17.994039    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls podName:f4425c17-5cc6-4afc-922c-d74694805196 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:21.994010965 +0000 UTC m=+9.008252323 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-g8wz8" (UID: "f4425c17-5cc6-4afc-922c-d74694805196") : secret "cluster-monitoring-operator-tls" not found
Nov 03 01:51:17.994417 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:17.994084    5005 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 01:51:17.994417 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:17.994099    5005 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bcfc5667d-w8v7m: secret "image-registry-tls" not found
Nov 03 01:51:17.994417 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:17.994085    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:17.994417 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:17.994145    5005 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 01:51:17.994417 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:17.994155    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls podName:108d9224-ccd3-4925-a65b-0d59e938ff72 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:21.99413973 +0000 UTC m=+9.008381088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls") pod "image-registry-6bcfc5667d-w8v7m" (UID: "108d9224-ccd3-4925-a65b-0d59e938ff72") : secret "image-registry-tls" not found
Nov 03 01:51:17.994417 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:17.994206    5005 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 01:51:17.994417 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:17.994240    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls podName:be449663-2797-487f-8559-0a96c563ae3a nodeName:}" failed. No retries permitted until 2026-11-03 01:51:21.994220049 +0000 UTC m=+9.008461405 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9twrb" (UID: "be449663-2797-487f-8559-0a96c563ae3a") : secret "insights-runtime-extractor-tls" not found
Nov 03 01:51:17.994417 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:17.994304    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-55bnn\" (UID: \"0110b28b-ec04-4f65-bc3d-2e82f88b30d7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn"
Nov 03 01:51:17.994417 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:17.994327    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:17.994417 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:17.994410    5005 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 01:51:17.994901 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:17.994431    5005 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 01:51:17.994901 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:17.994444    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls podName:2555270f-a2ea-4d11-8dff-d130f9786e3d nodeName:}" failed. No retries permitted until 2026-11-03 01:51:21.994433513 +0000 UTC m=+9.008674870 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls") pod "dns-default-w4h82" (UID: "2555270f-a2ea-4d11-8dff-d130f9786e3d") : secret "dns-default-metrics-tls" not found
Nov 03 01:51:17.994901 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:17.994479    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert podName:0110b28b-ec04-4f65-bc3d-2e82f88b30d7 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:21.994465795 +0000 UTC m=+9.008707153 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-55bnn" (UID: "0110b28b-ec04-4f65-bc3d-2e82f88b30d7") : secret "networking-console-plugin-cert" not found
Nov 03 01:51:17.994901 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:17.994489    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-service-ca-bundle\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:17.996425 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:17.996400    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a9e57fd-a6e6-4ddb-aa08-ac1390403544-metrics-certs\") pod \"router-default-7875dd7849-ptjkr\" (UID: \"7a9e57fd-a6e6-4ddb-aa08-ac1390403544\") " pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:17.996563 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:17.996542    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef90f36a-279e-4bfa-a97d-70132a266670-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-qtpnh\" (UID: \"ef90f36a-279e-4bfa-a97d-70132a266670\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:18.110546 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:18.110518    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:18.110673 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:18.110585    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"
Nov 03 01:51:18.529088 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:18.529056    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh"]
Nov 03 01:51:18.534138 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:18.534114    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7875dd7849-ptjkr"]
Nov 03 01:51:18.586908 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:18.586884    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9e57fd_a6e6_4ddb_aa08_ac1390403544.slice/crio-04b3ff924dfd4e3d1324a55a3c20c50210da3cca91008cb9f1541d31557df9dc WatchSource:0}: Error finding container 04b3ff924dfd4e3d1324a55a3c20c50210da3cca91008cb9f1541d31557df9dc: Status 404 returned error can't find the container with id 04b3ff924dfd4e3d1324a55a3c20c50210da3cca91008cb9f1541d31557df9dc
Nov 03 01:51:19.192088 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:19.192048    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7875dd7849-ptjkr" event={"ID":"7a9e57fd-a6e6-4ddb-aa08-ac1390403544","Type":"ContainerStarted","Data":"5338dd6e5ae614840078f25abde3446d095c1c4f245e4ddaf1f558ac34a112a6"}
Nov 03 01:51:19.192286 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:19.192129    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7875dd7849-ptjkr" event={"ID":"7a9e57fd-a6e6-4ddb-aa08-ac1390403544","Type":"ContainerStarted","Data":"04b3ff924dfd4e3d1324a55a3c20c50210da3cca91008cb9f1541d31557df9dc"}
Nov 03 01:51:19.193245 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:19.193217    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh" event={"ID":"ef90f36a-279e-4bfa-a97d-70132a266670","Type":"ContainerStarted","Data":"be77f4254e4ff280c7aee7df9e61c621b2d4ffe3a0c54c761e7a664d97eaa35b"}
Nov 03 01:51:20.110675 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:20.110634    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:20.113936 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:20.113915    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:20.638797 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:20.638772    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:20.767975 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:20.767942    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:20.768361 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:20.768343    5005 scope.go:122] "RemoveContainer" containerID="621ffc29ff310c22fafdd982e60f04f27b6d2cbafd590f6936927afc32e6259c"
Nov 03 01:51:20.768554 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:20.768532    5005 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rjmbd_openshift-console-operator(a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd" podUID="a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd"
Nov 03 01:51:20.926192 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:20.926050    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:21.182145 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:21.182074    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jdk59"
Nov 03 01:51:21.199443 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:21.199418    5005 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 01:51:22.032798 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:22.032759    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:22.032798 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:22.032801    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:22.032992 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:22.032908    5005 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 01:51:22.032992 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:22.032930    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:22.032992 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:22.032979    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-55bnn\" (UID: \"0110b28b-ec04-4f65-bc3d-2e82f88b30d7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn"
Nov 03 01:51:22.033153 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:22.033002    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:22.033153 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:22.033028    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs\") pod \"network-metrics-daemon-xv8zr\" (UID: \"c7a013c2-18a4-4372-adcb-8dfafa673f9c\") " pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:51:22.033153 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:22.033048    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:22.033153 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:22.032937    5005 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55fd5cff67-bb76l: secret "image-registry-tls" not found
Nov 03 01:51:22.033153 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:22.033135    5005 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 01:51:22.033153 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:22.033140    5005 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 01:51:22.033380 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:22.032908    5005 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 01:51:22.033380 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:22.033187    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls podName:9c546cea-b474-44cf-84a5-b89784bda1d6 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:30.033163319 +0000 UTC m=+17.047404685 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls") pod "image-registry-55fd5cff67-bb76l" (UID: "9c546cea-b474-44cf-84a5-b89784bda1d6") : secret "image-registry-tls" not found
Nov 03 01:51:22.033380 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:22.033192    5005 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bcfc5667d-w8v7m: secret "image-registry-tls" not found
Nov 03 01:51:22.033380 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:22.033209    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls podName:f4425c17-5cc6-4afc-922c-d74694805196 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:30.033198465 +0000 UTC m=+17.047439813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-g8wz8" (UID: "f4425c17-5cc6-4afc-922c-d74694805196") : secret "cluster-monitoring-operator-tls" not found
Nov 03 01:51:22.033380 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:22.033224    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs podName:c7a013c2-18a4-4372-adcb-8dfafa673f9c nodeName:}" failed. No retries permitted until 2026-11-03 01:51:30.03321605 +0000 UTC m=+17.047457397 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs") pod "network-metrics-daemon-xv8zr" (UID: "c7a013c2-18a4-4372-adcb-8dfafa673f9c") : secret "metrics-daemon-secret" not found
Nov 03 01:51:22.033380 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:22.033245    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls podName:108d9224-ccd3-4925-a65b-0d59e938ff72 nodeName:}" failed. No retries permitted until 2026-11-03 01:51:30.03322981 +0000 UTC m=+17.047471162 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls") pod "image-registry-6bcfc5667d-w8v7m" (UID: "108d9224-ccd3-4925-a65b-0d59e938ff72") : secret "image-registry-tls" not found
Nov 03 01:51:22.035398 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:22.035373    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be449663-2797-487f-8559-0a96c563ae3a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9twrb\" (UID: \"be449663-2797-487f-8559-0a96c563ae3a\") " pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:22.035398 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:22.035382    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2555270f-a2ea-4d11-8dff-d130f9786e3d-metrics-tls\") pod \"dns-default-w4h82\" (UID: \"2555270f-a2ea-4d11-8dff-d130f9786e3d\") " pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:22.035549 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:22.035526    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0110b28b-ec04-4f65-bc3d-2e82f88b30d7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-55bnn\" (UID: \"0110b28b-ec04-4f65-bc3d-2e82f88b30d7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn"
Nov 03 01:51:22.041312 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:22.041294    5005 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:22.041742 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:22.041725    5005 scope.go:122] "RemoveContainer" containerID="621ffc29ff310c22fafdd982e60f04f27b6d2cbafd590f6936927afc32e6259c"
Nov 03 01:51:22.041941 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:22.041917    5005 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-rjmbd_openshift-console-operator(a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd" podUID="a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd"
Nov 03 01:51:22.205369 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:22.205326    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh" event={"ID":"ef90f36a-279e-4bfa-a97d-70132a266670","Type":"ContainerStarted","Data":"d8795db3144ffef5095545742ec4329a76c437f7cefe63e7b31299d06b1fa55e"}
Nov 03 01:51:22.205839 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:22.205391    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-qtpnh" event={"ID":"ef90f36a-279e-4bfa-a97d-70132a266670","Type":"ContainerStarted","Data":"7ec29fbd6ff1e1ea43924d26b7d0499d1c952e203aae3f0cbee7986dc249ced3"}
Nov 03 01:51:22.313639 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:22.313556    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn"
Nov 03 01:51:22.324856 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:22.324830    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:22.329880 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:22.329854    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9twrb"
Nov 03 01:51:22.689375 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:22.689293    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-55bnn"]
Nov 03 01:51:22.739981 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:22.739953    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0110b28b_ec04_4f65_bc3d_2e82f88b30d7.slice/crio-cd77e8ec42496eb3b6c9b5401e2374f840fea21e4310c634f98307d2c2a22519 WatchSource:0}: Error finding container cd77e8ec42496eb3b6c9b5401e2374f840fea21e4310c634f98307d2c2a22519: Status 404 returned error can't find the container with id cd77e8ec42496eb3b6c9b5401e2374f840fea21e4310c634f98307d2c2a22519
Nov 03 01:51:22.742300 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:22.742278    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9twrb"]
Nov 03 01:51:22.748977 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:22.748957    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe449663_2797_487f_8559_0a96c563ae3a.slice/crio-8fcf2be79c722c28d1b287f25494d14b3053845b8fe4a7d42655cefecccdf004 WatchSource:0}: Error finding container 8fcf2be79c722c28d1b287f25494d14b3053845b8fe4a7d42655cefecccdf004: Status 404 returned error can't find the container with id 8fcf2be79c722c28d1b287f25494d14b3053845b8fe4a7d42655cefecccdf004
Nov 03 01:51:22.749319 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:22.749297    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w4h82"]
Nov 03 01:51:22.790504 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:22.790481    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2555270f_a2ea_4d11_8dff_d130f9786e3d.slice/crio-6d012bb0b8006b41d4e434251361992cf8bb7cdf151e64164cff2eb2e9287299 WatchSource:0}: Error finding container 6d012bb0b8006b41d4e434251361992cf8bb7cdf151e64164cff2eb2e9287299: Status 404 returned error can't find the container with id 6d012bb0b8006b41d4e434251361992cf8bb7cdf151e64164cff2eb2e9287299
Nov 03 01:51:23.209685 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:23.209648    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9twrb" event={"ID":"be449663-2797-487f-8559-0a96c563ae3a","Type":"ContainerStarted","Data":"039ede57eba630629093e16dad9beb07e19ec837e908e18c8a362a15de3c0c20"}
Nov 03 01:51:23.210084 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:23.209730    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9twrb" event={"ID":"be449663-2797-487f-8559-0a96c563ae3a","Type":"ContainerStarted","Data":"8fcf2be79c722c28d1b287f25494d14b3053845b8fe4a7d42655cefecccdf004"}
Nov 03 01:51:23.211215 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:23.211190    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w4h82" event={"ID":"2555270f-a2ea-4d11-8dff-d130f9786e3d","Type":"ContainerStarted","Data":"6d012bb0b8006b41d4e434251361992cf8bb7cdf151e64164cff2eb2e9287299"}
Nov 03 01:51:23.212310 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:23.212285    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn" event={"ID":"0110b28b-ec04-4f65-bc3d-2e82f88b30d7","Type":"ContainerStarted","Data":"cd77e8ec42496eb3b6c9b5401e2374f840fea21e4310c634f98307d2c2a22519"}
Nov 03 01:51:25.018536 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:25.018494    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:25.019041 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:25.018684    5005 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 01:51:25.020057 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:25.020038    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7875dd7849-ptjkr"
Nov 03 01:51:25.102415 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:25.102386    5005 scope.go:122] "RemoveContainer" containerID="621ffc29ff310c22fafdd982e60f04f27b6d2cbafd590f6936927afc32e6259c"
Nov 03 01:51:25.223475 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:25.223439    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-55bnn" event={"ID":"0110b28b-ec04-4f65-bc3d-2e82f88b30d7","Type":"ContainerStarted","Data":"ffc1075094dfda2d5c1b67adf76acadbadb7881f617b63e353f250ff606c8008"}
Nov 03 01:51:25.225489 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:25.225469    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 01:51:25.225625 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:25.225553    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd" event={"ID":"a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd","Type":"ContainerStarted","Data":"0aab9047bde7b8b864992e98efe0b108ea8752e39486ec1a492ec86c3e2530aa"}
Nov 03 01:51:25.225973 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:25.225949    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:25.226868 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:25.226754    5005 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-rjmbd container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.19:8443/readyz\": dial tcp 10.132.0.19:8443: connect: connection refused" start-of-body=
Nov 03 01:51:25.226868 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:25.226810    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd" podUID="a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.19:8443/readyz\": dial tcp 10.132.0.19:8443: connect: connection refused"
Nov 03 01:51:25.227600 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:25.227576    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9twrb" event={"ID":"be449663-2797-487f-8559-0a96c563ae3a","Type":"ContainerStarted","Data":"9477b001c2d890b6be283ec40e3b6dff57fe78bfd0d4e80c45cc70e74784fc46"}
Nov 03 01:51:26.233274 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:26.233232    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w4h82" event={"ID":"2555270f-a2ea-4d11-8dff-d130f9786e3d","Type":"ContainerStarted","Data":"f8db28c80390337b3cd6ba6c30d76a35a5c8bc45988abf5c2b83f98d828c2b1c"}
Nov 03 01:51:26.233737 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:26.233332    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w4h82" event={"ID":"2555270f-a2ea-4d11-8dff-d130f9786e3d","Type":"ContainerStarted","Data":"3f13b9fe0ab8ec44f5add0a7ee37827eccc729b6b4f188a5df0c73f643be3911"}
Nov 03 01:51:26.238534 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:26.238487    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-rjmbd"
Nov 03 01:51:27.239107 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:27.239071    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9twrb" event={"ID":"be449663-2797-487f-8559-0a96c563ae3a","Type":"ContainerStarted","Data":"cc63e2aaf62effa193ab826581aea4805bbf663e48c65f7e1f2f71c65de05f96"}
Nov 03 01:51:27.239578 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:27.239558    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:30.104755 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:30.104717    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:30.104755 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:30.104756    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:30.105275 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:30.104937    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs\") pod \"network-metrics-daemon-xv8zr\" (UID: \"c7a013c2-18a4-4372-adcb-8dfafa673f9c\") " pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:51:30.105275 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:30.104974    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:30.107138 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:30.107113    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a013c2-18a4-4372-adcb-8dfafa673f9c-metrics-certs\") pod \"network-metrics-daemon-xv8zr\" (UID: \"c7a013c2-18a4-4372-adcb-8dfafa673f9c\") " pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:51:30.107281 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:30.107155    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls\") pod \"image-registry-55fd5cff67-bb76l\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") " pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:30.107359 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:30.107335    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4425c17-5cc6-4afc-922c-d74694805196-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-g8wz8\" (UID: \"f4425c17-5cc6-4afc-922c-d74694805196\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:30.107404 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:30.107389    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls\") pod \"image-registry-6bcfc5667d-w8v7m\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") " pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:30.112234 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:30.112215    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"
Nov 03 01:51:30.114361 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:30.114340    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:30.114462 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:30.114403    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xv8zr"
Nov 03 01:51:30.124736 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:30.124720    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:30.522375 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:30.522342    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8"]
Nov 03 01:51:30.537188 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:30.537153    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4425c17_5cc6_4afc_922c_d74694805196.slice/crio-1cbc7a555c6221ccf2402231863c12fe3f15320e3e0cb34becce33d331b31e60 WatchSource:0}: Error finding container 1cbc7a555c6221ccf2402231863c12fe3f15320e3e0cb34becce33d331b31e60: Status 404 returned error can't find the container with id 1cbc7a555c6221ccf2402231863c12fe3f15320e3e0cb34becce33d331b31e60
Nov 03 01:51:30.568877 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:30.568851    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bcfc5667d-w8v7m"]
Nov 03 01:51:30.587679 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:30.587658    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108d9224_ccd3_4925_a65b_0d59e938ff72.slice/crio-d882e79ae5ad0c125d2829faa1f518de102e17a385a9e1586454a87c4af8faf1 WatchSource:0}: Error finding container d882e79ae5ad0c125d2829faa1f518de102e17a385a9e1586454a87c4af8faf1: Status 404 returned error can't find the container with id d882e79ae5ad0c125d2829faa1f518de102e17a385a9e1586454a87c4af8faf1
Nov 03 01:51:30.589958 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:30.589933    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xv8zr"]
Nov 03 01:51:30.601731 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:30.601684    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c546cea_b474_44cf_84a5_b89784bda1d6.slice/crio-82dee644bba88c80016217e48ba1c620cb878b66403228b0db7303c2a5a73f06 WatchSource:0}: Error finding container 82dee644bba88c80016217e48ba1c620cb878b66403228b0db7303c2a5a73f06: Status 404 returned error can't find the container with id 82dee644bba88c80016217e48ba1c620cb878b66403228b0db7303c2a5a73f06
Nov 03 01:51:30.615651 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:30.615629    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55fd5cff67-bb76l"]
Nov 03 01:51:30.716659 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:51:30.716580    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 01:51:31.254571 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:31.254530    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xv8zr" event={"ID":"c7a013c2-18a4-4372-adcb-8dfafa673f9c","Type":"ContainerStarted","Data":"cfddc260095ed14073cb84276226bef12de77300e6b82d8ff949dc048acda734"}
Nov 03 01:51:31.256133 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:31.256107    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m" event={"ID":"108d9224-ccd3-4925-a65b-0d59e938ff72","Type":"ContainerStarted","Data":"731ed6d4696e0afe660ad1556eb5fa20f43499c80628cde7af6a3cef78d78280"}
Nov 03 01:51:31.256260 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:31.256148    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m" event={"ID":"108d9224-ccd3-4925-a65b-0d59e938ff72","Type":"ContainerStarted","Data":"d882e79ae5ad0c125d2829faa1f518de102e17a385a9e1586454a87c4af8faf1"}
Nov 03 01:51:31.256320 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:31.256255    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:31.257634 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:31.257616    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55fd5cff67-bb76l" event={"ID":"9c546cea-b474-44cf-84a5-b89784bda1d6","Type":"ContainerStarted","Data":"885af36be776d77b382e620170286675d5642c82a9eb457b0927a7740a779b0d"}
Nov 03 01:51:31.257713 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:31.257648    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55fd5cff67-bb76l" event={"ID":"9c546cea-b474-44cf-84a5-b89784bda1d6","Type":"ContainerStarted","Data":"82dee644bba88c80016217e48ba1c620cb878b66403228b0db7303c2a5a73f06"}
Nov 03 01:51:31.257713 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:31.257675    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:31.258784 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:31.258764    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8" event={"ID":"f4425c17-5cc6-4afc-922c-d74694805196","Type":"ContainerStarted","Data":"1cbc7a555c6221ccf2402231863c12fe3f15320e3e0cb34becce33d331b31e60"}
Nov 03 01:51:33.268749 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:33.268717    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xv8zr" event={"ID":"c7a013c2-18a4-4372-adcb-8dfafa673f9c","Type":"ContainerStarted","Data":"efb8cda5ab094b6a90b590dd6141e3675cf696a68a12d6be86701e6aca494726"}
Nov 03 01:51:33.269200 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:33.268777    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xv8zr" event={"ID":"c7a013c2-18a4-4372-adcb-8dfafa673f9c","Type":"ContainerStarted","Data":"13df7ee856d4903318fab7f9dc8002124328453ef9be8e2d3eebd5276949dfac"}
Nov 03 01:51:34.273956 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:34.273920    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-g8wz8" event={"ID":"f4425c17-5cc6-4afc-922c-d74694805196","Type":"ContainerStarted","Data":"71c0d86828e90ee578931e5ba040edf3a8b8ec6b814518852dde29712a6c8b62"}
Nov 03 01:51:34.854943 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:34.854552    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-z27xq"]
Nov 03 01:51:34.863101 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:34.858170    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-z27xq"
Nov 03 01:51:34.878653 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:34.878627    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bcfc5667d-w8v7m"]
Nov 03 01:51:34.909027 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:34.908982    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 01:51:34.909273 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:34.908985    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 01:51:34.913830 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:34.913808    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-jrgr7\""
Nov 03 01:51:34.942091 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:34.942065    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-z27xq"]
Nov 03 01:51:34.947458 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:34.947427    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d038bf1c-df6d-400a-91c5-730b215bdba6-tmp\") pod \"downloads-6dd789c8b7-z27xq\" (UID: \"d038bf1c-df6d-400a-91c5-730b215bdba6\") " pod="openshift-console/downloads-6dd789c8b7-z27xq"
Nov 03 01:51:34.947762 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:34.947469    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgk4n\" (UniqueName: \"kubernetes.io/projected/d038bf1c-df6d-400a-91c5-730b215bdba6-kube-api-access-lgk4n\") pod \"downloads-6dd789c8b7-z27xq\" (UID: \"d038bf1c-df6d-400a-91c5-730b215bdba6\") " pod="openshift-console/downloads-6dd789c8b7-z27xq"
Nov 03 01:51:34.956449 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:34.956416    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75d5d488c8-f8s24"]
Nov 03 01:51:34.960282 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:34.960263    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.014278 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.014237    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75d5d488c8-f8s24"]
Nov 03 01:51:35.048004 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.047974    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-bound-sa-token\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.048141 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.048026    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-trusted-ca\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.048141 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.048104    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-ca-trust-extracted\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.048141 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.048136    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-registry-tls\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.048247 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.048210    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-registry-certificates\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.048294 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.048280    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-image-registry-private-configuration\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.048329 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.048316    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d038bf1c-df6d-400a-91c5-730b215bdba6-tmp\") pod \"downloads-6dd789c8b7-z27xq\" (UID: \"d038bf1c-df6d-400a-91c5-730b215bdba6\") " pod="openshift-console/downloads-6dd789c8b7-z27xq"
Nov 03 01:51:35.048361 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.048351    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r5k5\" (UniqueName: \"kubernetes.io/projected/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-kube-api-access-4r5k5\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.048394 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.048371    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgk4n\" (UniqueName: \"kubernetes.io/projected/d038bf1c-df6d-400a-91c5-730b215bdba6-kube-api-access-lgk4n\") pod \"downloads-6dd789c8b7-z27xq\" (UID: \"d038bf1c-df6d-400a-91c5-730b215bdba6\") " pod="openshift-console/downloads-6dd789c8b7-z27xq"
Nov 03 01:51:35.048394 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.048391    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-installation-pull-secrets\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.048630 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.048613    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d038bf1c-df6d-400a-91c5-730b215bdba6-tmp\") pod \"downloads-6dd789c8b7-z27xq\" (UID: \"d038bf1c-df6d-400a-91c5-730b215bdba6\") " pod="openshift-console/downloads-6dd789c8b7-z27xq"
Nov 03 01:51:35.102320 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.102294    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgk4n\" (UniqueName: \"kubernetes.io/projected/d038bf1c-df6d-400a-91c5-730b215bdba6-kube-api-access-lgk4n\") pod \"downloads-6dd789c8b7-z27xq\" (UID: \"d038bf1c-df6d-400a-91c5-730b215bdba6\") " pod="openshift-console/downloads-6dd789c8b7-z27xq"
Nov 03 01:51:35.149257 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.149181    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-ca-trust-extracted\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.149257 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.149231    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-registry-tls\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.149475 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.149259    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-registry-certificates\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.149475 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.149305    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-image-registry-private-configuration\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.149475 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.149362    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r5k5\" (UniqueName: \"kubernetes.io/projected/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-kube-api-access-4r5k5\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.149475 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.149395    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-installation-pull-secrets\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.149688 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.149633    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-ca-trust-extracted\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.149787 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.149716    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-bound-sa-token\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.149787 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.149763    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-trusted-ca\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.150237 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.150210    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-registry-certificates\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.151030 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.151005    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-trusted-ca\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.151895 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.151875    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-installation-pull-secrets\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.151973 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.151962    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-registry-tls\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.152068 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.152051    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-image-registry-private-configuration\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.172938 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.172909    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-z27xq"
Nov 03 01:51:35.202426 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.202398    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-bound-sa-token\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.202751 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.202733    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r5k5\" (UniqueName: \"kubernetes.io/projected/0a83921e-8b8f-4bbc-95c5-2a1dc33a2484-kube-api-access-4r5k5\") pod \"image-registry-75d5d488c8-f8s24\" (UID: \"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484\") " pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.271102 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.271068    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:35.488326 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.488256    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2jlbf"]
Nov 03 01:51:35.492599 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.492582    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2jlbf"
Nov 03 01:51:35.545527 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.545500    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2jlbf"]
Nov 03 01:51:35.549251 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.549226    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 03 01:51:35.549251 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.549243    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-cjgvc\""
Nov 03 01:51:35.553566 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.553543    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5e41d0c6-c6ea-4033-a362-98fca79bdd39-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2jlbf\" (UID: \"5e41d0c6-c6ea-4033-a362-98fca79bdd39\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2jlbf"
Nov 03 01:51:35.590091 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.590068    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-z27xq"]
Nov 03 01:51:35.604455 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:35.604431    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd038bf1c_df6d_400a_91c5_730b215bdba6.slice/crio-ae0b7145db7d4a90b7192a04f9c43929b3985b2469159dc8c75384ade442bf7a WatchSource:0}: Error finding container ae0b7145db7d4a90b7192a04f9c43929b3985b2469159dc8c75384ade442bf7a: Status 404 returned error can't find the container with id ae0b7145db7d4a90b7192a04f9c43929b3985b2469159dc8c75384ade442bf7a
Nov 03 01:51:35.645427 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.645396    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75d5d488c8-f8s24"]
Nov 03 01:51:35.653087 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:35.653065    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a83921e_8b8f_4bbc_95c5_2a1dc33a2484.slice/crio-c509022589fb3d05af7ee592b32de0eef726b62c4a26cbc91ff311c254d1cad9 WatchSource:0}: Error finding container c509022589fb3d05af7ee592b32de0eef726b62c4a26cbc91ff311c254d1cad9: Status 404 returned error can't find the container with id c509022589fb3d05af7ee592b32de0eef726b62c4a26cbc91ff311c254d1cad9
Nov 03 01:51:35.654054 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.654035    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5e41d0c6-c6ea-4033-a362-98fca79bdd39-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2jlbf\" (UID: \"5e41d0c6-c6ea-4033-a362-98fca79bdd39\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2jlbf"
Nov 03 01:51:35.656514 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.656494    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5e41d0c6-c6ea-4033-a362-98fca79bdd39-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2jlbf\" (UID: \"5e41d0c6-c6ea-4033-a362-98fca79bdd39\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2jlbf"
Nov 03 01:51:35.804386 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:35.804350    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2jlbf"
Nov 03 01:51:36.177872 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:36.177795    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2jlbf"]
Nov 03 01:51:36.185427 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:36.185392    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e41d0c6_c6ea_4033_a362_98fca79bdd39.slice/crio-73a3ff607404dec1e6cc2227a2201b7b4cbe060c179822c440e1177d6ecde38b WatchSource:0}: Error finding container 73a3ff607404dec1e6cc2227a2201b7b4cbe060c179822c440e1177d6ecde38b: Status 404 returned error can't find the container with id 73a3ff607404dec1e6cc2227a2201b7b4cbe060c179822c440e1177d6ecde38b
Nov 03 01:51:36.282483 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:36.282446    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2jlbf" event={"ID":"5e41d0c6-c6ea-4033-a362-98fca79bdd39","Type":"ContainerStarted","Data":"73a3ff607404dec1e6cc2227a2201b7b4cbe060c179822c440e1177d6ecde38b"}
Nov 03 01:51:36.283580 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:36.283557    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-z27xq" event={"ID":"d038bf1c-df6d-400a-91c5-730b215bdba6","Type":"ContainerStarted","Data":"ae0b7145db7d4a90b7192a04f9c43929b3985b2469159dc8c75384ade442bf7a"}
Nov 03 01:51:36.285060 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:36.285034    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75d5d488c8-f8s24" event={"ID":"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484","Type":"ContainerStarted","Data":"65b16115ac01ebb35155f15d1592e6ce0097048e4185cda343882e91885adca7"}
Nov 03 01:51:36.285177 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:36.285073    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75d5d488c8-f8s24" event={"ID":"0a83921e-8b8f-4bbc-95c5-2a1dc33a2484","Type":"ContainerStarted","Data":"c509022589fb3d05af7ee592b32de0eef726b62c4a26cbc91ff311c254d1cad9"}
Nov 03 01:51:36.285226 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:36.285171    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:36.396341 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:36.396135    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75d5d488c8-f8s24" podStartSLOduration=2.396116549 podStartE2EDuration="2.396116549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:36.395458163 +0000 UTC m=+23.409699529" watchObservedRunningTime="2026-11-03 01:51:36.396116549 +0000 UTC m=+23.410357915"
Nov 03 01:51:37.244949 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:37.244917    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w4h82"
Nov 03 01:51:38.295655 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:38.295616    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2jlbf" event={"ID":"5e41d0c6-c6ea-4033-a362-98fca79bdd39","Type":"ContainerStarted","Data":"3df81bef880f0d47412b41cd214e017463062dc0d9b8fc993b37bc7e5ed29113"}
Nov 03 01:51:38.296184 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:38.295835    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2jlbf"
Nov 03 01:51:38.301292 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:38.301269    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2jlbf"
Nov 03 01:51:38.402459 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:38.402346    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2jlbf" podStartSLOduration=1.491422965 podStartE2EDuration="3.402329802s" totalImagesPullingTime="1.910906837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:38.401042293 +0000 UTC m=+25.415283659" watchObservedRunningTime="2026-11-03 01:51:38.402329802 +0000 UTC m=+25.416571190"
Nov 03 01:51:38.482888 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:38.482858    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vvzws"
Nov 03 01:51:46.402470 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.402435    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xl2vn"]
Nov 03 01:51:46.432433 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.432395    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.566160 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.566124    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/069f1d59-3556-40b1-9769-f137ef715740-node-exporter-tls\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.566360 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.566175    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/069f1d59-3556-40b1-9769-f137ef715740-root\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.566360 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.566232    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/069f1d59-3556-40b1-9769-f137ef715740-node-exporter-wtmp\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.566360 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.566351    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/069f1d59-3556-40b1-9769-f137ef715740-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.566511 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.566388    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/069f1d59-3556-40b1-9769-f137ef715740-node-exporter-accelerators-collector-config\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.566511 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.566416    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/069f1d59-3556-40b1-9769-f137ef715740-metrics-client-ca\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.566615 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.566573    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/069f1d59-3556-40b1-9769-f137ef715740-node-exporter-textfile\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.566670 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.566615    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snj9l\" (UniqueName: \"kubernetes.io/projected/069f1d59-3556-40b1-9769-f137ef715740-kube-api-access-snj9l\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.566760 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.566735    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/069f1d59-3556-40b1-9769-f137ef715740-sys\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.585076 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.585025    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 01:51:46.585076 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.585073    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 01:51:46.585321 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.585152    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 01:51:46.585321 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.585149    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8rp9w\""
Nov 03 01:51:46.585321 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.585251    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 01:51:46.667537 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.667450    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/069f1d59-3556-40b1-9769-f137ef715740-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.667537 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.667508    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/069f1d59-3556-40b1-9769-f137ef715740-node-exporter-accelerators-collector-config\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.667537 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.667528    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/069f1d59-3556-40b1-9769-f137ef715740-metrics-client-ca\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.667871 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.667563    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/069f1d59-3556-40b1-9769-f137ef715740-node-exporter-textfile\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.667871 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.667590    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snj9l\" (UniqueName: \"kubernetes.io/projected/069f1d59-3556-40b1-9769-f137ef715740-kube-api-access-snj9l\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.667871 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.667633    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/069f1d59-3556-40b1-9769-f137ef715740-sys\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.667871 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.667665    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/069f1d59-3556-40b1-9769-f137ef715740-node-exporter-tls\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.667871 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.667688    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/069f1d59-3556-40b1-9769-f137ef715740-root\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.667871 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.667725    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/069f1d59-3556-40b1-9769-f137ef715740-node-exporter-wtmp\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.667871 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.667875    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/069f1d59-3556-40b1-9769-f137ef715740-node-exporter-wtmp\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.668160 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.667991    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/069f1d59-3556-40b1-9769-f137ef715740-sys\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.668308 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.668283    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/069f1d59-3556-40b1-9769-f137ef715740-metrics-client-ca\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.668413 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.668283    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/069f1d59-3556-40b1-9769-f137ef715740-node-exporter-accelerators-collector-config\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.668413 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.668333    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/069f1d59-3556-40b1-9769-f137ef715740-root\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.668606 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.668525    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/069f1d59-3556-40b1-9769-f137ef715740-node-exporter-textfile\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.670356 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.670331    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/069f1d59-3556-40b1-9769-f137ef715740-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.670446 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.670404    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/069f1d59-3556-40b1-9769-f137ef715740-node-exporter-tls\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.721840 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.721801    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snj9l\" (UniqueName: \"kubernetes.io/projected/069f1d59-3556-40b1-9769-f137ef715740-kube-api-access-snj9l\") pod \"node-exporter-xl2vn\" (UID: \"069f1d59-3556-40b1-9769-f137ef715740\") " pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.808630 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:46.808597    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xl2vn"
Nov 03 01:51:46.848667 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:51:46.848630    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069f1d59_3556_40b1_9769_f137ef715740.slice/crio-9ae29f138498a7b2fbd10ff9ad89c157c7e6ef1a883a64049502bdf04adce38c WatchSource:0}: Error finding container 9ae29f138498a7b2fbd10ff9ad89c157c7e6ef1a883a64049502bdf04adce38c: Status 404 returned error can't find the container with id 9ae29f138498a7b2fbd10ff9ad89c157c7e6ef1a883a64049502bdf04adce38c
Nov 03 01:51:47.287767 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:47.287728    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mdr8p"
Nov 03 01:51:47.340566 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:47.340523    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xl2vn" event={"ID":"069f1d59-3556-40b1-9769-f137ef715740","Type":"ContainerStarted","Data":"9ae29f138498a7b2fbd10ff9ad89c157c7e6ef1a883a64049502bdf04adce38c"}
Nov 03 01:51:51.362524 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:51.362482    5005 generic.go:356] "Generic (PLEG): container finished" podID="069f1d59-3556-40b1-9769-f137ef715740" containerID="6f2cb778803ab5e22c590af523967bbd4e5ebf479a636a8c3079aa843a641761" exitCode=0
Nov 03 01:51:51.363117 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:51.362545    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xl2vn" event={"ID":"069f1d59-3556-40b1-9769-f137ef715740","Type":"ContainerDied","Data":"6f2cb778803ab5e22c590af523967bbd4e5ebf479a636a8c3079aa843a641761"}
Nov 03 01:51:52.268440 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:52.268411    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:51:52.370046 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:52.370003    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xl2vn" event={"ID":"069f1d59-3556-40b1-9769-f137ef715740","Type":"ContainerStarted","Data":"16d88b92a0c204a535253428a70388b0879b8aaf2a466520357773cf6d18cfc5"}
Nov 03 01:51:52.370505 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:52.370083    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xl2vn" event={"ID":"069f1d59-3556-40b1-9769-f137ef715740","Type":"ContainerStarted","Data":"ee5aa7360a61513309c5e0525e554d408f6b440452a79f0079e535a40dc090cd"}
Nov 03 01:51:52.372206 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:52.372165    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-z27xq" event={"ID":"d038bf1c-df6d-400a-91c5-730b215bdba6","Type":"ContainerStarted","Data":"26a2c0ef2b9f2420e4dd0f60c3fc29e0bc15fa9b8e5e02094253d6ad674e4bc9"}
Nov 03 01:51:52.372878 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:52.372854    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-z27xq"
Nov 03 01:51:52.379142 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:52.379111    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-z27xq"
Nov 03 01:51:52.486079 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:52.485970    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xl2vn" podStartSLOduration=2.800143276 podStartE2EDuration="6.485951159s" totalImagesPullingTime="3.606478416s" totalInitContainerRuntime="79.329467ms" isStatefulPod=true podCreationTimestamp="2026-11-03 01:51:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:52.485433645 +0000 UTC m=+39.499675016" watchObservedRunningTime="2026-11-03 01:51:52.485951159 +0000 UTC m=+39.500192506"
Nov 03 01:51:52.594060 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:52.593994    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-z27xq" podStartSLOduration=2.052767067 podStartE2EDuration="18.593976829s" totalImagesPullingTime="16.541209762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:52.592664244 +0000 UTC m=+39.606905610" watchObservedRunningTime="2026-11-03 01:51:52.593976829 +0000 UTC m=+39.608218195"
Nov 03 01:51:54.885802 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:54.885714    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:51:57.294793 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:57.294759    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75d5d488c8-f8s24"
Nov 03 01:51:57.502421 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:57.502378    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55fd5cff67-bb76l"]
Nov 03 01:51:59.899384 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:51:59.899343    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m" podUID="108d9224-ccd3-4925-a65b-0d59e938ff72" containerName="registry" containerID="cri-o://731ed6d4696e0afe660ad1556eb5fa20f43499c80628cde7af6a3cef78d78280" gracePeriod=30
Nov 03 01:52:00.202177 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.202144    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:52:00.396318 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.396278    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls\") pod \"108d9224-ccd3-4925-a65b-0d59e938ff72\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") "
Nov 03 01:52:00.396504 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.396332    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/108d9224-ccd3-4925-a65b-0d59e938ff72-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/108d9224-ccd3-4925-a65b-0d59e938ff72-trusted-ca\") pod \"108d9224-ccd3-4925-a65b-0d59e938ff72\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") "
Nov 03 01:52:00.396504 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.396366    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-kube-api-access-cg5wd\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-kube-api-access-cg5wd\") pod \"108d9224-ccd3-4925-a65b-0d59e938ff72\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") "
Nov 03 01:52:00.396504 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.396422    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/108d9224-ccd3-4925-a65b-0d59e938ff72-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/108d9224-ccd3-4925-a65b-0d59e938ff72-installation-pull-secrets\") pod \"108d9224-ccd3-4925-a65b-0d59e938ff72\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") "
Nov 03 01:52:00.396504 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.396459    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108d9224-ccd3-4925-a65b-0d59e938ff72-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/108d9224-ccd3-4925-a65b-0d59e938ff72-ca-trust-extracted\") pod \"108d9224-ccd3-4925-a65b-0d59e938ff72\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") "
Nov 03 01:52:00.396749 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.396505    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-bound-sa-token\") pod \"108d9224-ccd3-4925-a65b-0d59e938ff72\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") "
Nov 03 01:52:00.396749 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.396575    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/108d9224-ccd3-4925-a65b-0d59e938ff72-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/108d9224-ccd3-4925-a65b-0d59e938ff72-image-registry-private-configuration\") pod \"108d9224-ccd3-4925-a65b-0d59e938ff72\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") "
Nov 03 01:52:00.396749 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.396594    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-certificates\") pod \"108d9224-ccd3-4925-a65b-0d59e938ff72\" (UID: \"108d9224-ccd3-4925-a65b-0d59e938ff72\") "
Nov 03 01:52:00.397296 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.396951    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/108d9224-ccd3-4925-a65b-0d59e938ff72-trusted-ca" pod "108d9224-ccd3-4925-a65b-0d59e938ff72" (UID: "108d9224-ccd3-4925-a65b-0d59e938ff72"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:52:00.397296 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.397217    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-certificates" pod "108d9224-ccd3-4925-a65b-0d59e938ff72" (UID: "108d9224-ccd3-4925-a65b-0d59e938ff72"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:52:00.400120 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.400091    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-kube-api-access-cg5wd" pod "108d9224-ccd3-4925-a65b-0d59e938ff72" (UID: "108d9224-ccd3-4925-a65b-0d59e938ff72"). InnerVolumeSpecName "kube-api-access-cg5wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:52:00.401177 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.401135    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/108d9224-ccd3-4925-a65b-0d59e938ff72-installation-pull-secrets" pod "108d9224-ccd3-4925-a65b-0d59e938ff72" (UID: "108d9224-ccd3-4925-a65b-0d59e938ff72"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:52:00.401634 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.401575    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-bound-sa-token" pod "108d9224-ccd3-4925-a65b-0d59e938ff72" (UID: "108d9224-ccd3-4925-a65b-0d59e938ff72"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:52:00.402813 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.402786    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls" pod "108d9224-ccd3-4925-a65b-0d59e938ff72" (UID: "108d9224-ccd3-4925-a65b-0d59e938ff72"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:52:00.403224 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.403190    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/108d9224-ccd3-4925-a65b-0d59e938ff72-image-registry-private-configuration" pod "108d9224-ccd3-4925-a65b-0d59e938ff72" (UID: "108d9224-ccd3-4925-a65b-0d59e938ff72"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:52:00.405493 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.405459    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108d9224-ccd3-4925-a65b-0d59e938ff72-ca-trust-extracted" pod "108d9224-ccd3-4925-a65b-0d59e938ff72" (UID: "108d9224-ccd3-4925-a65b-0d59e938ff72"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:00.414683 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.414614    5005 generic.go:356] "Generic (PLEG): container finished" podID="108d9224-ccd3-4925-a65b-0d59e938ff72" containerID="731ed6d4696e0afe660ad1556eb5fa20f43499c80628cde7af6a3cef78d78280" exitCode=0
Nov 03 01:52:00.414910 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.414688    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m"
Nov 03 01:52:00.415124 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.414721    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m" event={"ID":"108d9224-ccd3-4925-a65b-0d59e938ff72","Type":"ContainerDied","Data":"731ed6d4696e0afe660ad1556eb5fa20f43499c80628cde7af6a3cef78d78280"}
Nov 03 01:52:00.415209 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.415178    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bcfc5667d-w8v7m" event={"ID":"108d9224-ccd3-4925-a65b-0d59e938ff72","Type":"ContainerDied","Data":"d882e79ae5ad0c125d2829faa1f518de102e17a385a9e1586454a87c4af8faf1"}
Nov 03 01:52:00.415268 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.415231    5005 scope.go:122] "RemoveContainer" containerID="731ed6d4696e0afe660ad1556eb5fa20f43499c80628cde7af6a3cef78d78280"
Nov 03 01:52:00.426735 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.426685    5005 scope.go:122] "RemoveContainer" containerID="731ed6d4696e0afe660ad1556eb5fa20f43499c80628cde7af6a3cef78d78280"
Nov 03 01:52:00.427228 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:52:00.427198    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"731ed6d4696e0afe660ad1556eb5fa20f43499c80628cde7af6a3cef78d78280\": container with ID starting with 731ed6d4696e0afe660ad1556eb5fa20f43499c80628cde7af6a3cef78d78280 not found: ID does not exist" containerID="731ed6d4696e0afe660ad1556eb5fa20f43499c80628cde7af6a3cef78d78280"
Nov 03 01:52:00.427372 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.427238    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"731ed6d4696e0afe660ad1556eb5fa20f43499c80628cde7af6a3cef78d78280"} err="failed to get container status \"731ed6d4696e0afe660ad1556eb5fa20f43499c80628cde7af6a3cef78d78280\": rpc error: code = NotFound desc = could not find container \"731ed6d4696e0afe660ad1556eb5fa20f43499c80628cde7af6a3cef78d78280\": container with ID starting with 731ed6d4696e0afe660ad1556eb5fa20f43499c80628cde7af6a3cef78d78280 not found: ID does not exist"
Nov 03 01:52:00.498133 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.498095    5005 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/108d9224-ccd3-4925-a65b-0d59e938ff72-image-registry-private-configuration\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:00.498133 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.498131    5005 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-certificates\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:00.498133 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.498143    5005 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-registry-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:00.498397 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.498152    5005 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/108d9224-ccd3-4925-a65b-0d59e938ff72-trusted-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:00.498397 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.498161    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg5wd\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-kube-api-access-cg5wd\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:00.498397 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.498169    5005 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/108d9224-ccd3-4925-a65b-0d59e938ff72-installation-pull-secrets\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:00.498397 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.498178    5005 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/108d9224-ccd3-4925-a65b-0d59e938ff72-ca-trust-extracted\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:00.498397 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.498187    5005 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/108d9224-ccd3-4925-a65b-0d59e938ff72-bound-sa-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:00.570848 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.570802    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bcfc5667d-w8v7m"]
Nov 03 01:52:00.574149 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:00.574123    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6bcfc5667d-w8v7m"]
Nov 03 01:52:00.716983 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:52:00.716909    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 01:52:01.109576 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:01.109544    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="108d9224-ccd3-4925-a65b-0d59e938ff72" path="/var/lib/kubelet/pods/108d9224-ccd3-4925-a65b-0d59e938ff72/volumes"
Nov 03 01:52:22.527391 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.527331    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-55fd5cff67-bb76l" podUID="9c546cea-b474-44cf-84a5-b89784bda1d6" containerName="registry" containerID="cri-o://885af36be776d77b382e620170286675d5642c82a9eb457b0927a7740a779b0d" gracePeriod=30
Nov 03 01:52:22.810913 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.810881    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:52:22.889504 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.889473    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-kube-api-access-85997\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-kube-api-access-85997\") pod \"9c546cea-b474-44cf-84a5-b89784bda1d6\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") "
Nov 03 01:52:22.889720 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.889514    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c546cea-b474-44cf-84a5-b89784bda1d6-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c546cea-b474-44cf-84a5-b89784bda1d6-installation-pull-secrets\") pod \"9c546cea-b474-44cf-84a5-b89784bda1d6\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") "
Nov 03 01:52:22.889720 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.889540    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls\") pod \"9c546cea-b474-44cf-84a5-b89784bda1d6\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") "
Nov 03 01:52:22.889720 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.889569    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c546cea-b474-44cf-84a5-b89784bda1d6-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c546cea-b474-44cf-84a5-b89784bda1d6-image-registry-private-configuration\") pod \"9c546cea-b474-44cf-84a5-b89784bda1d6\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") "
Nov 03 01:52:22.889720 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.889641    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c546cea-b474-44cf-84a5-b89784bda1d6-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c546cea-b474-44cf-84a5-b89784bda1d6-ca-trust-extracted\") pod \"9c546cea-b474-44cf-84a5-b89784bda1d6\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") "
Nov 03 01:52:22.889720 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.889719    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c546cea-b474-44cf-84a5-b89784bda1d6-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c546cea-b474-44cf-84a5-b89784bda1d6-trusted-ca\") pod \"9c546cea-b474-44cf-84a5-b89784bda1d6\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") "
Nov 03 01:52:22.889989 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.889764    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-bound-sa-token\") pod \"9c546cea-b474-44cf-84a5-b89784bda1d6\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") "
Nov 03 01:52:22.889989 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.889801    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-certificates\") pod \"9c546cea-b474-44cf-84a5-b89784bda1d6\" (UID: \"9c546cea-b474-44cf-84a5-b89784bda1d6\") "
Nov 03 01:52:22.890550 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.890370    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-certificates" pod "9c546cea-b474-44cf-84a5-b89784bda1d6" (UID: "9c546cea-b474-44cf-84a5-b89784bda1d6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:52:22.890550 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.890427    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c546cea-b474-44cf-84a5-b89784bda1d6-trusted-ca" pod "9c546cea-b474-44cf-84a5-b89784bda1d6" (UID: "9c546cea-b474-44cf-84a5-b89784bda1d6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:52:22.892062 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.892028    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-kube-api-access-85997" pod "9c546cea-b474-44cf-84a5-b89784bda1d6" (UID: "9c546cea-b474-44cf-84a5-b89784bda1d6"). InnerVolumeSpecName "kube-api-access-85997". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:52:22.892175 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.892090    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls" pod "9c546cea-b474-44cf-84a5-b89784bda1d6" (UID: "9c546cea-b474-44cf-84a5-b89784bda1d6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:52:22.892260 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.892233    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c546cea-b474-44cf-84a5-b89784bda1d6-installation-pull-secrets" pod "9c546cea-b474-44cf-84a5-b89784bda1d6" (UID: "9c546cea-b474-44cf-84a5-b89784bda1d6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:52:22.892368 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.892308    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c546cea-b474-44cf-84a5-b89784bda1d6-image-registry-private-configuration" pod "9c546cea-b474-44cf-84a5-b89784bda1d6" (UID: "9c546cea-b474-44cf-84a5-b89784bda1d6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:52:22.892635 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.892610    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-bound-sa-token" pod "9c546cea-b474-44cf-84a5-b89784bda1d6" (UID: "9c546cea-b474-44cf-84a5-b89784bda1d6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:52:22.896840 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.896814    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c546cea-b474-44cf-84a5-b89784bda1d6-ca-trust-extracted" pod "9c546cea-b474-44cf-84a5-b89784bda1d6" (UID: "9c546cea-b474-44cf-84a5-b89784bda1d6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:22.990896 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.990864    5005 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-bound-sa-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:22.990896 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.990895    5005 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-certificates\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:22.991091 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.990906    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85997\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-kube-api-access-85997\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:22.991091 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.990915    5005 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c546cea-b474-44cf-84a5-b89784bda1d6-installation-pull-secrets\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:22.991091 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.990924    5005 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c546cea-b474-44cf-84a5-b89784bda1d6-registry-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:22.991091 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.990934    5005 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c546cea-b474-44cf-84a5-b89784bda1d6-image-registry-private-configuration\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:22.991091 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.990943    5005 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c546cea-b474-44cf-84a5-b89784bda1d6-ca-trust-extracted\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:22.991091 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:22.990952    5005 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c546cea-b474-44cf-84a5-b89784bda1d6-trusted-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:23.507629 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:23.507594    5005 generic.go:356] "Generic (PLEG): container finished" podID="9c546cea-b474-44cf-84a5-b89784bda1d6" containerID="885af36be776d77b382e620170286675d5642c82a9eb457b0927a7740a779b0d" exitCode=0
Nov 03 01:52:23.507842 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:23.507645    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55fd5cff67-bb76l"
Nov 03 01:52:23.507842 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:23.507684    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55fd5cff67-bb76l" event={"ID":"9c546cea-b474-44cf-84a5-b89784bda1d6","Type":"ContainerDied","Data":"885af36be776d77b382e620170286675d5642c82a9eb457b0927a7740a779b0d"}
Nov 03 01:52:23.507842 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:23.507743    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55fd5cff67-bb76l" event={"ID":"9c546cea-b474-44cf-84a5-b89784bda1d6","Type":"ContainerDied","Data":"82dee644bba88c80016217e48ba1c620cb878b66403228b0db7303c2a5a73f06"}
Nov 03 01:52:23.507842 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:23.507763    5005 scope.go:122] "RemoveContainer" containerID="885af36be776d77b382e620170286675d5642c82a9eb457b0927a7740a779b0d"
Nov 03 01:52:23.515828 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:23.515809    5005 scope.go:122] "RemoveContainer" containerID="885af36be776d77b382e620170286675d5642c82a9eb457b0927a7740a779b0d"
Nov 03 01:52:23.516085 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:52:23.516066    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"885af36be776d77b382e620170286675d5642c82a9eb457b0927a7740a779b0d\": container with ID starting with 885af36be776d77b382e620170286675d5642c82a9eb457b0927a7740a779b0d not found: ID does not exist" containerID="885af36be776d77b382e620170286675d5642c82a9eb457b0927a7740a779b0d"
Nov 03 01:52:23.516135 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:23.516091    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"885af36be776d77b382e620170286675d5642c82a9eb457b0927a7740a779b0d"} err="failed to get container status \"885af36be776d77b382e620170286675d5642c82a9eb457b0927a7740a779b0d\": rpc error: code = NotFound desc = could not find container \"885af36be776d77b382e620170286675d5642c82a9eb457b0927a7740a779b0d\": container with ID starting with 885af36be776d77b382e620170286675d5642c82a9eb457b0927a7740a779b0d not found: ID does not exist"
Nov 03 01:52:23.707007 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:23.706970    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55fd5cff67-bb76l"]
Nov 03 01:52:23.753348 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:23.753318    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-55fd5cff67-bb76l"]
Nov 03 01:52:25.106746 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:25.106713    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c546cea-b474-44cf-84a5-b89784bda1d6" path="/var/lib/kubelet/pods/9c546cea-b474-44cf-84a5-b89784bda1d6/volumes"
Nov 03 01:52:25.518011 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:25.517976    5005 generic.go:356] "Generic (PLEG): container finished" podID="baf5281f-5938-4417-b22f-2644034da7ef" containerID="7ecc92e2b116eace93766511ee1bfb6ebdf024d8dba845a6151a2b2fbd3a92e5" exitCode=0
Nov 03 01:52:25.518186 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:25.518044    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr" event={"ID":"baf5281f-5938-4417-b22f-2644034da7ef","Type":"ContainerDied","Data":"7ecc92e2b116eace93766511ee1bfb6ebdf024d8dba845a6151a2b2fbd3a92e5"}
Nov 03 01:52:25.518387 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:25.518374    5005 scope.go:122] "RemoveContainer" containerID="7ecc92e2b116eace93766511ee1bfb6ebdf024d8dba845a6151a2b2fbd3a92e5"
Nov 03 01:52:26.523404 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:26.523366    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-nw7sr" event={"ID":"baf5281f-5938-4417-b22f-2644034da7ef","Type":"ContainerStarted","Data":"058fe4a0142777f7af58c7ab4ee26fef224723a897030bfbb933dda37a5e220b"}
Nov 03 01:52:30.711362 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:52:30.711325    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 01:52:35.559117 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:35.559074    5005 generic.go:356] "Generic (PLEG): container finished" podID="15face97-689e-4fef-86c0-17a2a85829db" containerID="e07ebe8dc2dfdac20310e43c2ebbb4d7ff208cbac7997abc252073c579cd57fd" exitCode=0
Nov 03 01:52:35.559546 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:35.559127    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn" event={"ID":"15face97-689e-4fef-86c0-17a2a85829db","Type":"ContainerDied","Data":"e07ebe8dc2dfdac20310e43c2ebbb4d7ff208cbac7997abc252073c579cd57fd"}
Nov 03 01:52:35.559546 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:35.559429    5005 scope.go:122] "RemoveContainer" containerID="e07ebe8dc2dfdac20310e43c2ebbb4d7ff208cbac7997abc252073c579cd57fd"
Nov 03 01:52:36.564917 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:36.564884    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t59nn" event={"ID":"15face97-689e-4fef-86c0-17a2a85829db","Type":"ContainerStarted","Data":"0ac840141b3e5f2cb452057291304fa5f7cf3a47a643ad97cd1eaec31c047c5c"}
Nov 03 01:52:40.581112 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:40.581077    5005 generic.go:356] "Generic (PLEG): container finished" podID="354d7018-887a-497f-b74f-195c8c88e184" containerID="463d876c65e5e5075dfd59f6b8c35ea5f48d96b38817b9dfa7b504c63b6ba219" exitCode=0
Nov 03 01:52:40.581516 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:40.581150    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-gqz7v" event={"ID":"354d7018-887a-497f-b74f-195c8c88e184","Type":"ContainerDied","Data":"463d876c65e5e5075dfd59f6b8c35ea5f48d96b38817b9dfa7b504c63b6ba219"}
Nov 03 01:52:40.581561 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:40.581513    5005 scope.go:122] "RemoveContainer" containerID="463d876c65e5e5075dfd59f6b8c35ea5f48d96b38817b9dfa7b504c63b6ba219"
Nov 03 01:52:41.586660 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:41.586624    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-gqz7v" event={"ID":"354d7018-887a-497f-b74f-195c8c88e184","Type":"ContainerStarted","Data":"3b6ff8d7e956499f939f432ac1681ced97855b6e2b7802f4e8213a6a9404e5b4"}
Nov 03 01:52:41.608419 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:41.608390    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-gxg2j-p5824"]
Nov 03 01:52:41.608588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:41.608569    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-gxg2j-p5824" podUID="62c3ca06-b2a4-49d7-84db-1390277be2c8" containerName="insights-gathering" containerID="cri-o://d3f053ab4b69e0277c6fa3e29443517b0e6657a53305ff725e86df346681d4eb" gracePeriod=30
Nov 03 01:52:41.858516 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:41.858493    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:52:41.955370 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:41.955340    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c3ca06-b2a4-49d7-84db-1390277be2c8-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62c3ca06-b2a4-49d7-84db-1390277be2c8-service-ca-bundle\") pod \"62c3ca06-b2a4-49d7-84db-1390277be2c8\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") "
Nov 03 01:52:41.955523 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:41.955404    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c3ca06-b2a4-49d7-84db-1390277be2c8-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62c3ca06-b2a4-49d7-84db-1390277be2c8-trusted-ca-bundle\") pod \"62c3ca06-b2a4-49d7-84db-1390277be2c8\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") "
Nov 03 01:52:41.955523 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:41.955441    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c3ca06-b2a4-49d7-84db-1390277be2c8-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/62c3ca06-b2a4-49d7-84db-1390277be2c8-archives-path\") pod \"62c3ca06-b2a4-49d7-84db-1390277be2c8\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") "
Nov 03 01:52:41.955523 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:41.955490    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c3ca06-b2a4-49d7-84db-1390277be2c8-kube-api-access-z46t7\" (UniqueName: \"kubernetes.io/projected/62c3ca06-b2a4-49d7-84db-1390277be2c8-kube-api-access-z46t7\") pod \"62c3ca06-b2a4-49d7-84db-1390277be2c8\" (UID: \"62c3ca06-b2a4-49d7-84db-1390277be2c8\") "
Nov 03 01:52:41.955815 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:41.955788    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c3ca06-b2a4-49d7-84db-1390277be2c8-archives-path" pod "62c3ca06-b2a4-49d7-84db-1390277be2c8" (UID: "62c3ca06-b2a4-49d7-84db-1390277be2c8"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:41.955815 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:41.955800    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c3ca06-b2a4-49d7-84db-1390277be2c8-service-ca-bundle" pod "62c3ca06-b2a4-49d7-84db-1390277be2c8" (UID: "62c3ca06-b2a4-49d7-84db-1390277be2c8"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:52:41.955975 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:41.955825    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c3ca06-b2a4-49d7-84db-1390277be2c8-trusted-ca-bundle" pod "62c3ca06-b2a4-49d7-84db-1390277be2c8" (UID: "62c3ca06-b2a4-49d7-84db-1390277be2c8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:52:41.957785 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:41.957754    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c3ca06-b2a4-49d7-84db-1390277be2c8-kube-api-access-z46t7" pod "62c3ca06-b2a4-49d7-84db-1390277be2c8" (UID: "62c3ca06-b2a4-49d7-84db-1390277be2c8"). InnerVolumeSpecName "kube-api-access-z46t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:52:42.056467 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:42.056436    5005 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62c3ca06-b2a4-49d7-84db-1390277be2c8-trusted-ca-bundle\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:42.056467 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:42.056462    5005 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/62c3ca06-b2a4-49d7-84db-1390277be2c8-archives-path\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:42.056467 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:42.056471    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z46t7\" (UniqueName: \"kubernetes.io/projected/62c3ca06-b2a4-49d7-84db-1390277be2c8-kube-api-access-z46t7\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:42.056688 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:42.056481    5005 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62c3ca06-b2a4-49d7-84db-1390277be2c8-service-ca-bundle\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:42.592185 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:42.592150    5005 generic.go:356] "Generic (PLEG): container finished" podID="62c3ca06-b2a4-49d7-84db-1390277be2c8" containerID="d3f053ab4b69e0277c6fa3e29443517b0e6657a53305ff725e86df346681d4eb" exitCode=2
Nov 03 01:52:42.592664 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:42.592199    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gxg2j-p5824"
Nov 03 01:52:42.592664 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:42.592230    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gxg2j-p5824" event={"ID":"62c3ca06-b2a4-49d7-84db-1390277be2c8","Type":"ContainerDied","Data":"d3f053ab4b69e0277c6fa3e29443517b0e6657a53305ff725e86df346681d4eb"}
Nov 03 01:52:42.592664 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:42.592280    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gxg2j-p5824" event={"ID":"62c3ca06-b2a4-49d7-84db-1390277be2c8","Type":"ContainerDied","Data":"e59b0b9e967622046c45b1bbd1249b6ea96e8186fc6340e94dd34ee27174e7ac"}
Nov 03 01:52:42.592664 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:42.592299    5005 scope.go:122] "RemoveContainer" containerID="d3f053ab4b69e0277c6fa3e29443517b0e6657a53305ff725e86df346681d4eb"
Nov 03 01:52:42.601263 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:42.601244    5005 scope.go:122] "RemoveContainer" containerID="d3f053ab4b69e0277c6fa3e29443517b0e6657a53305ff725e86df346681d4eb"
Nov 03 01:52:42.601557 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:52:42.601539    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f053ab4b69e0277c6fa3e29443517b0e6657a53305ff725e86df346681d4eb\": container with ID starting with d3f053ab4b69e0277c6fa3e29443517b0e6657a53305ff725e86df346681d4eb not found: ID does not exist" containerID="d3f053ab4b69e0277c6fa3e29443517b0e6657a53305ff725e86df346681d4eb"
Nov 03 01:52:42.601619 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:42.601571    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f053ab4b69e0277c6fa3e29443517b0e6657a53305ff725e86df346681d4eb"} err="failed to get container status \"d3f053ab4b69e0277c6fa3e29443517b0e6657a53305ff725e86df346681d4eb\": rpc error: code = NotFound desc = could not find container \"d3f053ab4b69e0277c6fa3e29443517b0e6657a53305ff725e86df346681d4eb\": container with ID starting with d3f053ab4b69e0277c6fa3e29443517b0e6657a53305ff725e86df346681d4eb not found: ID does not exist"
Nov 03 01:52:42.745167 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:42.745137    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-gxg2j-p5824"]
Nov 03 01:52:42.791270 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:42.791241    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-gxg2j-p5824"]
Nov 03 01:52:43.106398 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:52:43.106356    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c3ca06-b2a4-49d7-84db-1390277be2c8" path="/var/lib/kubelet/pods/62c3ca06-b2a4-49d7-84db-1390277be2c8/volumes"
Nov 03 01:53:00.713550 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:53:00.713516    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 01:53:30.712338 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:53:30.712256    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 01:54:00.712035 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:54:00.711997    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 01:54:28.729604 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.729568    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh"]
Nov 03 01:54:28.730004 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.729901    5005 cpu_manager.go:414] "containerMap: removing container" podUID="9c546cea-b474-44cf-84a5-b89784bda1d6" containerName="registry"
Nov 03 01:54:28.730004 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.729912    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c546cea-b474-44cf-84a5-b89784bda1d6" containerName="registry"
Nov 03 01:54:28.730004 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.729929    5005 cpu_manager.go:414] "containerMap: removing container" podUID="62c3ca06-b2a4-49d7-84db-1390277be2c8" containerName="insights-gathering"
Nov 03 01:54:28.730004 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.729934    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c3ca06-b2a4-49d7-84db-1390277be2c8" containerName="insights-gathering"
Nov 03 01:54:28.730004 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.729958    5005 cpu_manager.go:414] "containerMap: removing container" podUID="108d9224-ccd3-4925-a65b-0d59e938ff72" containerName="registry"
Nov 03 01:54:28.730004 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.729964    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="108d9224-ccd3-4925-a65b-0d59e938ff72" containerName="registry"
Nov 03 01:54:28.730195 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.730009    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c546cea-b474-44cf-84a5-b89784bda1d6" containerName="registry"
Nov 03 01:54:28.730195 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.730021    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c3ca06-b2a4-49d7-84db-1390277be2c8" containerName="insights-gathering"
Nov 03 01:54:28.730195 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.730033    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="108d9224-ccd3-4925-a65b-0d59e938ff72" containerName="registry"
Nov 03 01:54:28.733202 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.733183    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh"
Nov 03 01:54:28.755529 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.755507    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz"]
Nov 03 01:54:28.758649 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.758634    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz"
Nov 03 01:54:28.781075 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.781052    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h"]
Nov 03 01:54:28.785502 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.785482    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h"
Nov 03 01:54:28.801057 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.801033    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-d6qwm\""
Nov 03 01:54:28.801057 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.801039    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 01:54:28.801177 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.801112    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 01:54:28.807388 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.807371    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9hdm\" (UniqueName: \"kubernetes.io/projected/e4197e24-8ed5-453a-8273-6471221f3385-kube-api-access-x9hdm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh\" (UID: \"e4197e24-8ed5-453a-8273-6471221f3385\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh"
Nov 03 01:54:28.807450 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.807412    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4197e24-8ed5-453a-8273-6471221f3385-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh\" (UID: \"e4197e24-8ed5-453a-8273-6471221f3385\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh"
Nov 03 01:54:28.807503 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.807484    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4197e24-8ed5-453a-8273-6471221f3385-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh\" (UID: \"e4197e24-8ed5-453a-8273-6471221f3385\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh"
Nov 03 01:54:28.807545 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.807532    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce68bd37-cfb9-44a5-8100-89db675d1265-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz\" (UID: \"ce68bd37-cfb9-44a5-8100-89db675d1265\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz"
Nov 03 01:54:28.807592 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.807555    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce68bd37-cfb9-44a5-8100-89db675d1265-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz\" (UID: \"ce68bd37-cfb9-44a5-8100-89db675d1265\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz"
Nov 03 01:54:28.807644 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.807630    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7j72\" (UniqueName: \"kubernetes.io/projected/ce68bd37-cfb9-44a5-8100-89db675d1265-kube-api-access-w7j72\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz\" (UID: \"ce68bd37-cfb9-44a5-8100-89db675d1265\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz"
Nov 03 01:54:28.827905 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.827881    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh"]
Nov 03 01:54:28.828974 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.828955    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m"]
Nov 03 01:54:28.832239 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.832224    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz"]
Nov 03 01:54:28.832321 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.832307    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m"
Nov 03 01:54:28.836842 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.836822    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h"]
Nov 03 01:54:28.873127 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.873105    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m"]
Nov 03 01:54:28.882496 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.882473    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-jw44s\""
Nov 03 01:54:28.882607 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.882569    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-f2486\""
Nov 03 01:54:28.882607 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.882575    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-dr55r\""
Nov 03 01:54:28.908253 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.908229    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4197e24-8ed5-453a-8273-6471221f3385-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh\" (UID: \"e4197e24-8ed5-453a-8273-6471221f3385\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh"
Nov 03 01:54:28.908341 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.908271    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69322313-2a08-49e8-9a58-6b06a89b79d4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h\" (UID: \"69322313-2a08-49e8-9a58-6b06a89b79d4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h"
Nov 03 01:54:28.908341 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.908293    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce68bd37-cfb9-44a5-8100-89db675d1265-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz\" (UID: \"ce68bd37-cfb9-44a5-8100-89db675d1265\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz"
Nov 03 01:54:28.908412 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.908351    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6knz\" (UniqueName: \"kubernetes.io/projected/69322313-2a08-49e8-9a58-6b06a89b79d4-kube-api-access-l6knz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h\" (UID: \"69322313-2a08-49e8-9a58-6b06a89b79d4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h"
Nov 03 01:54:28.908412 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.908385    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce68bd37-cfb9-44a5-8100-89db675d1265-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz\" (UID: \"ce68bd37-cfb9-44a5-8100-89db675d1265\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz"
Nov 03 01:54:28.908498 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.908414    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7j72\" (UniqueName: \"kubernetes.io/projected/ce68bd37-cfb9-44a5-8100-89db675d1265-kube-api-access-w7j72\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz\" (UID: \"ce68bd37-cfb9-44a5-8100-89db675d1265\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz"
Nov 03 01:54:28.908498 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.908438    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69322313-2a08-49e8-9a58-6b06a89b79d4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h\" (UID: \"69322313-2a08-49e8-9a58-6b06a89b79d4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h"
Nov 03 01:54:28.908498 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.908477    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wpt2\" (UniqueName: \"kubernetes.io/projected/e609f442-f2f5-49e4-99e1-05108a43d01f-kube-api-access-8wpt2\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m\" (UID: \"e609f442-f2f5-49e4-99e1-05108a43d01f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m"
Nov 03 01:54:28.908646 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.908505    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9hdm\" (UniqueName: \"kubernetes.io/projected/e4197e24-8ed5-453a-8273-6471221f3385-kube-api-access-x9hdm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh\" (UID: \"e4197e24-8ed5-453a-8273-6471221f3385\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh"
Nov 03 01:54:28.908646 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.908528    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e609f442-f2f5-49e4-99e1-05108a43d01f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m\" (UID: \"e609f442-f2f5-49e4-99e1-05108a43d01f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m"
Nov 03 01:54:28.908646 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.908639    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4197e24-8ed5-453a-8273-6471221f3385-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh\" (UID: \"e4197e24-8ed5-453a-8273-6471221f3385\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh"
Nov 03 01:54:28.908805 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.908652    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce68bd37-cfb9-44a5-8100-89db675d1265-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz\" (UID: \"ce68bd37-cfb9-44a5-8100-89db675d1265\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz"
Nov 03 01:54:28.908805 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.908646    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4197e24-8ed5-453a-8273-6471221f3385-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh\" (UID: \"e4197e24-8ed5-453a-8273-6471221f3385\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh"
Nov 03 01:54:28.908805 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.908733    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e609f442-f2f5-49e4-99e1-05108a43d01f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m\" (UID: \"e609f442-f2f5-49e4-99e1-05108a43d01f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m"
Nov 03 01:54:28.908911 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.908894    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4197e24-8ed5-453a-8273-6471221f3385-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh\" (UID: \"e4197e24-8ed5-453a-8273-6471221f3385\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh"
Nov 03 01:54:28.908949 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.908926    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce68bd37-cfb9-44a5-8100-89db675d1265-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz\" (UID: \"ce68bd37-cfb9-44a5-8100-89db675d1265\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz"
Nov 03 01:54:28.961213 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.961186    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9hdm\" (UniqueName: \"kubernetes.io/projected/e4197e24-8ed5-453a-8273-6471221f3385-kube-api-access-x9hdm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh\" (UID: \"e4197e24-8ed5-453a-8273-6471221f3385\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh"
Nov 03 01:54:28.961337 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:28.961268    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7j72\" (UniqueName: \"kubernetes.io/projected/ce68bd37-cfb9-44a5-8100-89db675d1265-kube-api-access-w7j72\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz\" (UID: \"ce68bd37-cfb9-44a5-8100-89db675d1265\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz"
Nov 03 01:54:29.009324 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:29.009297    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69322313-2a08-49e8-9a58-6b06a89b79d4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h\" (UID: \"69322313-2a08-49e8-9a58-6b06a89b79d4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h"
Nov 03 01:54:29.009419 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:29.009332    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6knz\" (UniqueName: \"kubernetes.io/projected/69322313-2a08-49e8-9a58-6b06a89b79d4-kube-api-access-l6knz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h\" (UID: \"69322313-2a08-49e8-9a58-6b06a89b79d4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h"
Nov 03 01:54:29.009419 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:29.009359    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69322313-2a08-49e8-9a58-6b06a89b79d4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h\" (UID: \"69322313-2a08-49e8-9a58-6b06a89b79d4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h"
Nov 03 01:54:29.009419 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:29.009384    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wpt2\" (UniqueName: \"kubernetes.io/projected/e609f442-f2f5-49e4-99e1-05108a43d01f-kube-api-access-8wpt2\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m\" (UID: \"e609f442-f2f5-49e4-99e1-05108a43d01f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m"
Nov 03 01:54:29.009419 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:29.009415    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e609f442-f2f5-49e4-99e1-05108a43d01f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m\" (UID: \"e609f442-f2f5-49e4-99e1-05108a43d01f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m"
Nov 03 01:54:29.009611 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:29.009459    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e609f442-f2f5-49e4-99e1-05108a43d01f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m\" (UID: \"e609f442-f2f5-49e4-99e1-05108a43d01f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m"
Nov 03 01:54:29.009776 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:29.009718    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69322313-2a08-49e8-9a58-6b06a89b79d4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h\" (UID: \"69322313-2a08-49e8-9a58-6b06a89b79d4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h"
Nov 03 01:54:29.009776 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:29.009769    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69322313-2a08-49e8-9a58-6b06a89b79d4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h\" (UID: \"69322313-2a08-49e8-9a58-6b06a89b79d4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h"
Nov 03 01:54:29.009903 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:29.009839    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e609f442-f2f5-49e4-99e1-05108a43d01f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m\" (UID: \"e609f442-f2f5-49e4-99e1-05108a43d01f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m"
Nov 03 01:54:29.009903 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:29.009893    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e609f442-f2f5-49e4-99e1-05108a43d01f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m\" (UID: \"e609f442-f2f5-49e4-99e1-05108a43d01f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m"
Nov 03 01:54:29.048435 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:29.048418    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh"
Nov 03 01:54:29.062845 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:29.062816    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6knz\" (UniqueName: \"kubernetes.io/projected/69322313-2a08-49e8-9a58-6b06a89b79d4-kube-api-access-l6knz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h\" (UID: \"69322313-2a08-49e8-9a58-6b06a89b79d4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h"
Nov 03 01:54:29.063023 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:29.063004    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wpt2\" (UniqueName: \"kubernetes.io/projected/e609f442-f2f5-49e4-99e1-05108a43d01f-kube-api-access-8wpt2\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m\" (UID: \"e609f442-f2f5-49e4-99e1-05108a43d01f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m"
Nov 03 01:54:29.073655 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:29.073635    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz"
Nov 03 01:54:29.103059 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:29.102842    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h"
Nov 03 01:54:29.150376 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:29.148604    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m"
Nov 03 01:54:29.451953 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:29.451878    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh"]
Nov 03 01:54:29.459007 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:54:29.458973    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4197e24_8ed5_453a_8273_6471221f3385.slice/crio-d0345d0f30a38f6519854fbdf800c1f062679e88300342ef751227ca01d07dc3 WatchSource:0}: Error finding container d0345d0f30a38f6519854fbdf800c1f062679e88300342ef751227ca01d07dc3: Status 404 returned error can't find the container with id d0345d0f30a38f6519854fbdf800c1f062679e88300342ef751227ca01d07dc3
Nov 03 01:54:29.470426 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:29.470402    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz"]
Nov 03 01:54:29.477377 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:54:29.477353    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce68bd37_cfb9_44a5_8100_89db675d1265.slice/crio-6510e966526c903ff79771938f064f3f8caa34c2cbd6732589273a0496a7258f WatchSource:0}: Error finding container 6510e966526c903ff79771938f064f3f8caa34c2cbd6732589273a0496a7258f: Status 404 returned error can't find the container with id 6510e966526c903ff79771938f064f3f8caa34c2cbd6732589273a0496a7258f
Nov 03 01:54:29.505792 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:29.505766    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h"]
Nov 03 01:54:29.512314 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:54:29.512291    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69322313_2a08_49e8_9a58_6b06a89b79d4.slice/crio-2b9bb5697b99b94627858c0aa89d8a4b92a0500d065cd0a2023527b47cda98e0 WatchSource:0}: Error finding container 2b9bb5697b99b94627858c0aa89d8a4b92a0500d065cd0a2023527b47cda98e0: Status 404 returned error can't find the container with id 2b9bb5697b99b94627858c0aa89d8a4b92a0500d065cd0a2023527b47cda98e0
Nov 03 01:54:29.523393 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:29.523371    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m"]
Nov 03 01:54:29.530254 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:54:29.530235    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode609f442_f2f5_49e4_99e1_05108a43d01f.slice/crio-2c95a8524b3c84e858164925386a095610ddf2815fbf7065c90eafb3fd318b13 WatchSource:0}: Error finding container 2c95a8524b3c84e858164925386a095610ddf2815fbf7065c90eafb3fd318b13: Status 404 returned error can't find the container with id 2c95a8524b3c84e858164925386a095610ddf2815fbf7065c90eafb3fd318b13
Nov 03 01:54:30.012398 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:30.012367    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m" event={"ID":"e609f442-f2f5-49e4-99e1-05108a43d01f","Type":"ContainerStarted","Data":"2c95a8524b3c84e858164925386a095610ddf2815fbf7065c90eafb3fd318b13"}
Nov 03 01:54:30.013491 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:30.013468    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz" event={"ID":"ce68bd37-cfb9-44a5-8100-89db675d1265","Type":"ContainerStarted","Data":"6510e966526c903ff79771938f064f3f8caa34c2cbd6732589273a0496a7258f"}
Nov 03 01:54:30.020343 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:30.020314    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh" event={"ID":"e4197e24-8ed5-453a-8273-6471221f3385","Type":"ContainerStarted","Data":"d0345d0f30a38f6519854fbdf800c1f062679e88300342ef751227ca01d07dc3"}
Nov 03 01:54:30.021470 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:30.021449    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h" event={"ID":"69322313-2a08-49e8-9a58-6b06a89b79d4","Type":"ContainerStarted","Data":"2b9bb5697b99b94627858c0aa89d8a4b92a0500d065cd0a2023527b47cda98e0"}
Nov 03 01:54:30.734229 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:54:30.734196    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 01:54:31.329679 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.329642    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"]
Nov 03 01:54:31.333766 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.333740    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.382390 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.382355    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-9cgjw\""
Nov 03 01:54:31.410435 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.410408    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 01:54:31.430653 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.430601    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"]
Nov 03 01:54:31.434790 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.434735    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.434930 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.434859    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.434930 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.434892    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.435040 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.434938    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.435040 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.435007    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tt2k\" (UniqueName: \"kubernetes.io/projected/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-kube-api-access-7tt2k\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.435142 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.435038    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.435142 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.435077    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.435213 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.435144    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.435213 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.435178    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.536076 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.536035    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.536279 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.536109    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.536279 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.536148    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.536279 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.536194    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.536279 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.536246    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.536279 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.536271    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.536461 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.536323    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.536461 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.536358    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tt2k\" (UniqueName: \"kubernetes.io/projected/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-kube-api-access-7tt2k\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.536461 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.536423    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.536933 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.536910    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.537045 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.537019    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.537107 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.537063    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.537280 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.537255    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.537362 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.537337    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.546533 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.546507    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.547027 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.547004    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.593932 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.593862    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tt2k\" (UniqueName: \"kubernetes.io/projected/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-kube-api-access-7tt2k\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.594519 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.594494    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ef5e81-eb41-4e31-a4d3-437e40ea8778-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4\" (UID: \"c7ef5e81-eb41-4e31-a4d3-437e40ea8778\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:31.654449 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:31.654415    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:32.118812 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:32.118716    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"]
Nov 03 01:54:32.166935 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:54:32.166900    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ef5e81_eb41_4e31_a4d3_437e40ea8778.slice/crio-f869c2f87d2de38b0f4d272cb45f211bf01e447ee6af4da7f9dd5ff4bb9f677e WatchSource:0}: Error finding container f869c2f87d2de38b0f4d272cb45f211bf01e447ee6af4da7f9dd5ff4bb9f677e: Status 404 returned error can't find the container with id f869c2f87d2de38b0f4d272cb45f211bf01e447ee6af4da7f9dd5ff4bb9f677e
Nov 03 01:54:33.041667 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:33.041627    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4" event={"ID":"c7ef5e81-eb41-4e31-a4d3-437e40ea8778","Type":"ContainerStarted","Data":"f869c2f87d2de38b0f4d272cb45f211bf01e447ee6af4da7f9dd5ff4bb9f677e"}
Nov 03 01:54:38.313083 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:38.312541    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 01:54:38.313083 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:38.312638    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 01:54:38.313083 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:38.312707    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 01:54:39.086084 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:39.086040    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4" event={"ID":"c7ef5e81-eb41-4e31-a4d3-437e40ea8778","Type":"ContainerStarted","Data":"603bbd1e24abfceb274d40c03836aaf78222ff738ce7080ae46309d9a8f4d40e"}
Nov 03 01:54:39.242742 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:39.242673    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4" podStartSLOduration=2.100312925 podStartE2EDuration="8.242657244s" totalImagesPullingTime="6.142344319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:54:39.240460669 +0000 UTC m=+206.254702038" watchObservedRunningTime="2026-11-03 01:54:39.242657244 +0000 UTC m=+206.256898615"
Nov 03 01:54:39.655419 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:39.655148    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:39.661379 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:39.661351    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:40.094307 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:40.094262    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h" event={"ID":"69322313-2a08-49e8-9a58-6b06a89b79d4","Type":"ContainerStarted","Data":"8324cd06a74d73da71e4ff6fbf00bb1aab2fb70a4248eef13567e9f5e0b03469"}
Nov 03 01:54:40.096490 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:40.096446    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m" event={"ID":"e609f442-f2f5-49e4-99e1-05108a43d01f","Type":"ContainerStarted","Data":"551cd864ba50c51b7d787015927b73e68cae86c81252ab3cff04edafad25ed95"}
Nov 03 01:54:40.098494 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:40.098455    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz" event={"ID":"ce68bd37-cfb9-44a5-8100-89db675d1265","Type":"ContainerStarted","Data":"d5f8085c75e3db70654f6163fc238f739b28350d88b3abda8bde57f5f7eb8139"}
Nov 03 01:54:40.100648 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:40.100579    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh" event={"ID":"e4197e24-8ed5-453a-8273-6471221f3385","Type":"ContainerStarted","Data":"fd5584475d847170e4b60e55fd9f07c3fa31113114087bc93a972a708ad6ffdb"}
Nov 03 01:54:40.101169 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:40.101144    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:40.102893 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:40.102856    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4"
Nov 03 01:54:41.105906 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:41.105871    5005 generic.go:356] "Generic (PLEG): container finished" podID="69322313-2a08-49e8-9a58-6b06a89b79d4" containerID="8324cd06a74d73da71e4ff6fbf00bb1aab2fb70a4248eef13567e9f5e0b03469" exitCode=0
Nov 03 01:54:41.107241 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:41.107212    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h" event={"ID":"69322313-2a08-49e8-9a58-6b06a89b79d4","Type":"ContainerDied","Data":"8324cd06a74d73da71e4ff6fbf00bb1aab2fb70a4248eef13567e9f5e0b03469"}
Nov 03 01:54:41.107610 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:41.107590    5005 generic.go:356] "Generic (PLEG): container finished" podID="e609f442-f2f5-49e4-99e1-05108a43d01f" containerID="551cd864ba50c51b7d787015927b73e68cae86c81252ab3cff04edafad25ed95" exitCode=0
Nov 03 01:54:41.107725 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:41.107659    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m" event={"ID":"e609f442-f2f5-49e4-99e1-05108a43d01f","Type":"ContainerDied","Data":"551cd864ba50c51b7d787015927b73e68cae86c81252ab3cff04edafad25ed95"}
Nov 03 01:54:41.109233 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:41.109208    5005 generic.go:356] "Generic (PLEG): container finished" podID="ce68bd37-cfb9-44a5-8100-89db675d1265" containerID="d5f8085c75e3db70654f6163fc238f739b28350d88b3abda8bde57f5f7eb8139" exitCode=0
Nov 03 01:54:41.109350 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:41.109282    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz" event={"ID":"ce68bd37-cfb9-44a5-8100-89db675d1265","Type":"ContainerDied","Data":"d5f8085c75e3db70654f6163fc238f739b28350d88b3abda8bde57f5f7eb8139"}
Nov 03 01:54:41.110954 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:41.110931    5005 generic.go:356] "Generic (PLEG): container finished" podID="e4197e24-8ed5-453a-8273-6471221f3385" containerID="fd5584475d847170e4b60e55fd9f07c3fa31113114087bc93a972a708ad6ffdb" exitCode=0
Nov 03 01:54:41.111045 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:41.111003    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh" event={"ID":"e4197e24-8ed5-453a-8273-6471221f3385","Type":"ContainerDied","Data":"fd5584475d847170e4b60e55fd9f07c3fa31113114087bc93a972a708ad6ffdb"}
Nov 03 01:54:43.121313 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:43.121230    5005 generic.go:356] "Generic (PLEG): container finished" podID="e609f442-f2f5-49e4-99e1-05108a43d01f" containerID="63a399179e4e9e95faaf0b3545bdd2f2b51fd035e44eef3994aa473744b39b82" exitCode=0
Nov 03 01:54:43.121313 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:43.121290    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m" event={"ID":"e609f442-f2f5-49e4-99e1-05108a43d01f","Type":"ContainerDied","Data":"63a399179e4e9e95faaf0b3545bdd2f2b51fd035e44eef3994aa473744b39b82"}
Nov 03 01:54:43.123185 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:43.123166    5005 generic.go:356] "Generic (PLEG): container finished" podID="ce68bd37-cfb9-44a5-8100-89db675d1265" containerID="9f232a7d70cf5708541f669ee7af6e02c1837080e2485435fbd13168d809c5c9" exitCode=0
Nov 03 01:54:43.123322 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:43.123232    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz" event={"ID":"ce68bd37-cfb9-44a5-8100-89db675d1265","Type":"ContainerDied","Data":"9f232a7d70cf5708541f669ee7af6e02c1837080e2485435fbd13168d809c5c9"}
Nov 03 01:54:43.125047 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:43.125028    5005 generic.go:356] "Generic (PLEG): container finished" podID="e4197e24-8ed5-453a-8273-6471221f3385" containerID="bfbca7993720cbee041a1f37db9f1c41ee4381cebff63a464f1fc6c517423024" exitCode=0
Nov 03 01:54:43.125144 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:43.125050    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh" event={"ID":"e4197e24-8ed5-453a-8273-6471221f3385","Type":"ContainerDied","Data":"bfbca7993720cbee041a1f37db9f1c41ee4381cebff63a464f1fc6c517423024"}
Nov 03 01:54:43.127443 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:43.127424    5005 generic.go:356] "Generic (PLEG): container finished" podID="69322313-2a08-49e8-9a58-6b06a89b79d4" containerID="67fb127cadd6ac63d27262328536267845702f92db4ec88f0c5d8e2661fa44b9" exitCode=0
Nov 03 01:54:43.127526 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:43.127444    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h" event={"ID":"69322313-2a08-49e8-9a58-6b06a89b79d4","Type":"ContainerDied","Data":"67fb127cadd6ac63d27262328536267845702f92db4ec88f0c5d8e2661fa44b9"}
Nov 03 01:54:53.195675 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:53.195634    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz" event={"ID":"ce68bd37-cfb9-44a5-8100-89db675d1265","Type":"ContainerStarted","Data":"b911301ed5855cb95fb9c362bf2399c09d68ccbfe95080034e74676157a1522a"}
Nov 03 01:54:53.198579 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:53.198550    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h" event={"ID":"69322313-2a08-49e8-9a58-6b06a89b79d4","Type":"ContainerStarted","Data":"6968f4e568f91ec1d4505c783e99e390a8c17126f4df351670597c4e3414edf6"}
Nov 03 01:54:53.333294 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:53.333181    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz" podStartSLOduration=3.113492085 podStartE2EDuration="25.333166738s" totalImagesPullingTime="22.219674653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:54:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:54:53.331525491 +0000 UTC m=+220.345766860" watchObservedRunningTime="2026-11-03 01:54:53.333166738 +0000 UTC m=+220.347408118"
Nov 03 01:54:53.440982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:53.440925    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h" podStartSLOduration=3.4881159090000002 podStartE2EDuration="25.440911029s" totalImagesPullingTime="21.95279512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:54:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:54:53.439400113 +0000 UTC m=+220.453641479" watchObservedRunningTime="2026-11-03 01:54:53.440911029 +0000 UTC m=+220.455152394"
Nov 03 01:54:54.205029 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:54.204995    5005 generic.go:356] "Generic (PLEG): container finished" podID="69322313-2a08-49e8-9a58-6b06a89b79d4" containerID="6968f4e568f91ec1d4505c783e99e390a8c17126f4df351670597c4e3414edf6" exitCode=0
Nov 03 01:54:54.205482 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:54.205082    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h" event={"ID":"69322313-2a08-49e8-9a58-6b06a89b79d4","Type":"ContainerDied","Data":"6968f4e568f91ec1d4505c783e99e390a8c17126f4df351670597c4e3414edf6"}
Nov 03 01:54:54.207017 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:54.206996    5005 generic.go:356] "Generic (PLEG): container finished" podID="e609f442-f2f5-49e4-99e1-05108a43d01f" containerID="2493902150f034ed3af86bbdd69f3972a3df1588284e9abfdd2841f919efed8c" exitCode=0
Nov 03 01:54:54.207138 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:54.207066    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m" event={"ID":"e609f442-f2f5-49e4-99e1-05108a43d01f","Type":"ContainerDied","Data":"2493902150f034ed3af86bbdd69f3972a3df1588284e9abfdd2841f919efed8c"}
Nov 03 01:54:54.208953 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:54.208933    5005 generic.go:356] "Generic (PLEG): container finished" podID="ce68bd37-cfb9-44a5-8100-89db675d1265" containerID="b911301ed5855cb95fb9c362bf2399c09d68ccbfe95080034e74676157a1522a" exitCode=0
Nov 03 01:54:54.209065 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:54.208957    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz" event={"ID":"ce68bd37-cfb9-44a5-8100-89db675d1265","Type":"ContainerDied","Data":"b911301ed5855cb95fb9c362bf2399c09d68ccbfe95080034e74676157a1522a"}
Nov 03 01:54:54.210895 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:54.210873    5005 generic.go:356] "Generic (PLEG): container finished" podID="e4197e24-8ed5-453a-8273-6471221f3385" containerID="9de51f8847fffc43ad0d73d1927ecb4d79603810181bcc6ece77ae50cbd17453" exitCode=0
Nov 03 01:54:54.210995 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:54.210949    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh" event={"ID":"e4197e24-8ed5-453a-8273-6471221f3385","Type":"ContainerDied","Data":"9de51f8847fffc43ad0d73d1927ecb4d79603810181bcc6ece77ae50cbd17453"}
Nov 03 01:54:55.370088 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.370058    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h"
Nov 03 01:54:55.406285 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.406261    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz"
Nov 03 01:54:55.409237 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.409219    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m"
Nov 03 01:54:55.412461 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.412442    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh"
Nov 03 01:54:55.469368 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.469312    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69322313-2a08-49e8-9a58-6b06a89b79d4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/69322313-2a08-49e8-9a58-6b06a89b79d4-bundle\") pod \"69322313-2a08-49e8-9a58-6b06a89b79d4\" (UID: \"69322313-2a08-49e8-9a58-6b06a89b79d4\") "
Nov 03 01:54:55.469368 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.469352    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69322313-2a08-49e8-9a58-6b06a89b79d4-kube-api-access-l6knz\" (UniqueName: \"kubernetes.io/projected/69322313-2a08-49e8-9a58-6b06a89b79d4-kube-api-access-l6knz\") pod \"69322313-2a08-49e8-9a58-6b06a89b79d4\" (UID: \"69322313-2a08-49e8-9a58-6b06a89b79d4\") "
Nov 03 01:54:55.469548 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.469382    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69322313-2a08-49e8-9a58-6b06a89b79d4-util\" (UniqueName: \"kubernetes.io/empty-dir/69322313-2a08-49e8-9a58-6b06a89b79d4-util\") pod \"69322313-2a08-49e8-9a58-6b06a89b79d4\" (UID: \"69322313-2a08-49e8-9a58-6b06a89b79d4\") "
Nov 03 01:54:55.469951 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.469927    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69322313-2a08-49e8-9a58-6b06a89b79d4-bundle" pod "69322313-2a08-49e8-9a58-6b06a89b79d4" (UID: "69322313-2a08-49e8-9a58-6b06a89b79d4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:55.471489 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.471469    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69322313-2a08-49e8-9a58-6b06a89b79d4-kube-api-access-l6knz" pod "69322313-2a08-49e8-9a58-6b06a89b79d4" (UID: "69322313-2a08-49e8-9a58-6b06a89b79d4"). InnerVolumeSpecName "kube-api-access-l6knz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:54:55.474751 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.474728    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69322313-2a08-49e8-9a58-6b06a89b79d4-util" pod "69322313-2a08-49e8-9a58-6b06a89b79d4" (UID: "69322313-2a08-49e8-9a58-6b06a89b79d4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:55.570709 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.570670    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4197e24-8ed5-453a-8273-6471221f3385-util\" (UniqueName: \"kubernetes.io/empty-dir/e4197e24-8ed5-453a-8273-6471221f3385-util\") pod \"e4197e24-8ed5-453a-8273-6471221f3385\" (UID: \"e4197e24-8ed5-453a-8273-6471221f3385\") "
Nov 03 01:54:55.570872 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.570716    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e609f442-f2f5-49e4-99e1-05108a43d01f-util\" (UniqueName: \"kubernetes.io/empty-dir/e609f442-f2f5-49e4-99e1-05108a43d01f-util\") pod \"e609f442-f2f5-49e4-99e1-05108a43d01f\" (UID: \"e609f442-f2f5-49e4-99e1-05108a43d01f\") "
Nov 03 01:54:55.570872 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.570743    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e609f442-f2f5-49e4-99e1-05108a43d01f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e609f442-f2f5-49e4-99e1-05108a43d01f-bundle\") pod \"e609f442-f2f5-49e4-99e1-05108a43d01f\" (UID: \"e609f442-f2f5-49e4-99e1-05108a43d01f\") "
Nov 03 01:54:55.570872 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.570759    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4197e24-8ed5-453a-8273-6471221f3385-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4197e24-8ed5-453a-8273-6471221f3385-bundle\") pod \"e4197e24-8ed5-453a-8273-6471221f3385\" (UID: \"e4197e24-8ed5-453a-8273-6471221f3385\") "
Nov 03 01:54:55.570872 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.570789    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4197e24-8ed5-453a-8273-6471221f3385-kube-api-access-x9hdm\" (UniqueName: \"kubernetes.io/projected/e4197e24-8ed5-453a-8273-6471221f3385-kube-api-access-x9hdm\") pod \"e4197e24-8ed5-453a-8273-6471221f3385\" (UID: \"e4197e24-8ed5-453a-8273-6471221f3385\") "
Nov 03 01:54:55.570872 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.570813    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce68bd37-cfb9-44a5-8100-89db675d1265-util\" (UniqueName: \"kubernetes.io/empty-dir/ce68bd37-cfb9-44a5-8100-89db675d1265-util\") pod \"ce68bd37-cfb9-44a5-8100-89db675d1265\" (UID: \"ce68bd37-cfb9-44a5-8100-89db675d1265\") "
Nov 03 01:54:55.570872 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.570870    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce68bd37-cfb9-44a5-8100-89db675d1265-kube-api-access-w7j72\" (UniqueName: \"kubernetes.io/projected/ce68bd37-cfb9-44a5-8100-89db675d1265-kube-api-access-w7j72\") pod \"ce68bd37-cfb9-44a5-8100-89db675d1265\" (UID: \"ce68bd37-cfb9-44a5-8100-89db675d1265\") "
Nov 03 01:54:55.571186 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.570916    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e609f442-f2f5-49e4-99e1-05108a43d01f-kube-api-access-8wpt2\" (UniqueName: \"kubernetes.io/projected/e609f442-f2f5-49e4-99e1-05108a43d01f-kube-api-access-8wpt2\") pod \"e609f442-f2f5-49e4-99e1-05108a43d01f\" (UID: \"e609f442-f2f5-49e4-99e1-05108a43d01f\") "
Nov 03 01:54:55.571186 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.570965    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce68bd37-cfb9-44a5-8100-89db675d1265-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce68bd37-cfb9-44a5-8100-89db675d1265-bundle\") pod \"ce68bd37-cfb9-44a5-8100-89db675d1265\" (UID: \"ce68bd37-cfb9-44a5-8100-89db675d1265\") "
Nov 03 01:54:55.571288 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.571231    5005 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69322313-2a08-49e8-9a58-6b06a89b79d4-bundle\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:55.571288 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.571251    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6knz\" (UniqueName: \"kubernetes.io/projected/69322313-2a08-49e8-9a58-6b06a89b79d4-kube-api-access-l6knz\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:55.571288 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.571266    5005 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69322313-2a08-49e8-9a58-6b06a89b79d4-util\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:55.571436 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.571333    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4197e24-8ed5-453a-8273-6471221f3385-bundle" pod "e4197e24-8ed5-453a-8273-6471221f3385" (UID: "e4197e24-8ed5-453a-8273-6471221f3385"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:55.571436 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.571345    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e609f442-f2f5-49e4-99e1-05108a43d01f-bundle" pod "e609f442-f2f5-49e4-99e1-05108a43d01f" (UID: "e609f442-f2f5-49e4-99e1-05108a43d01f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:55.571817 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.571791    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce68bd37-cfb9-44a5-8100-89db675d1265-bundle" pod "ce68bd37-cfb9-44a5-8100-89db675d1265" (UID: "ce68bd37-cfb9-44a5-8100-89db675d1265"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:55.573496 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.573465    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4197e24-8ed5-453a-8273-6471221f3385-kube-api-access-x9hdm" pod "e4197e24-8ed5-453a-8273-6471221f3385" (UID: "e4197e24-8ed5-453a-8273-6471221f3385"). InnerVolumeSpecName "kube-api-access-x9hdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:54:55.573650 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.573630    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e609f442-f2f5-49e4-99e1-05108a43d01f-kube-api-access-8wpt2" pod "e609f442-f2f5-49e4-99e1-05108a43d01f" (UID: "e609f442-f2f5-49e4-99e1-05108a43d01f"). InnerVolumeSpecName "kube-api-access-8wpt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:54:55.573915 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.573892    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce68bd37-cfb9-44a5-8100-89db675d1265-kube-api-access-w7j72" pod "ce68bd37-cfb9-44a5-8100-89db675d1265" (UID: "ce68bd37-cfb9-44a5-8100-89db675d1265"). InnerVolumeSpecName "kube-api-access-w7j72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:54:55.575244 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.575224    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e609f442-f2f5-49e4-99e1-05108a43d01f-util" pod "e609f442-f2f5-49e4-99e1-05108a43d01f" (UID: "e609f442-f2f5-49e4-99e1-05108a43d01f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:55.579217 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.579189    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4197e24-8ed5-453a-8273-6471221f3385-util" pod "e4197e24-8ed5-453a-8273-6471221f3385" (UID: "e4197e24-8ed5-453a-8273-6471221f3385"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:55.579487 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.579465    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce68bd37-cfb9-44a5-8100-89db675d1265-util" pod "ce68bd37-cfb9-44a5-8100-89db675d1265" (UID: "ce68bd37-cfb9-44a5-8100-89db675d1265"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:55.672387 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.672355    5005 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4197e24-8ed5-453a-8273-6471221f3385-util\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:55.672387 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.672383    5005 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e609f442-f2f5-49e4-99e1-05108a43d01f-util\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:55.672387 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.672392    5005 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e609f442-f2f5-49e4-99e1-05108a43d01f-bundle\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:55.672624 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.672404    5005 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4197e24-8ed5-453a-8273-6471221f3385-bundle\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:55.672624 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.672413    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9hdm\" (UniqueName: \"kubernetes.io/projected/e4197e24-8ed5-453a-8273-6471221f3385-kube-api-access-x9hdm\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:55.672624 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.672422    5005 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce68bd37-cfb9-44a5-8100-89db675d1265-util\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:55.672624 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.672430    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7j72\" (UniqueName: \"kubernetes.io/projected/ce68bd37-cfb9-44a5-8100-89db675d1265-kube-api-access-w7j72\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:55.672624 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.672440    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wpt2\" (UniqueName: \"kubernetes.io/projected/e609f442-f2f5-49e4-99e1-05108a43d01f-kube-api-access-8wpt2\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:55.672624 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:55.672450    5005 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce68bd37-cfb9-44a5-8100-89db675d1265-bundle\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:56.221050 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:56.221024    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h"
Nov 03 01:54:56.221050 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:56.221040    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8zk8h" event={"ID":"69322313-2a08-49e8-9a58-6b06a89b79d4","Type":"ContainerDied","Data":"2b9bb5697b99b94627858c0aa89d8a4b92a0500d065cd0a2023527b47cda98e0"}
Nov 03 01:54:56.221283 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:56.221090    5005 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b9bb5697b99b94627858c0aa89d8a4b92a0500d065cd0a2023527b47cda98e0"
Nov 03 01:54:56.222976 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:56.222952    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m" event={"ID":"e609f442-f2f5-49e4-99e1-05108a43d01f","Type":"ContainerDied","Data":"2c95a8524b3c84e858164925386a095610ddf2815fbf7065c90eafb3fd318b13"}
Nov 03 01:54:56.223099 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:56.222992    5005 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c95a8524b3c84e858164925386a095610ddf2815fbf7065c90eafb3fd318b13"
Nov 03 01:54:56.223099 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:56.222963    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mgm4m"
Nov 03 01:54:56.224816 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:56.224795    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz" event={"ID":"ce68bd37-cfb9-44a5-8100-89db675d1265","Type":"ContainerDied","Data":"6510e966526c903ff79771938f064f3f8caa34c2cbd6732589273a0496a7258f"}
Nov 03 01:54:56.224924 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:56.224822    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nxtmz"
Nov 03 01:54:56.224924 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:56.224830    5005 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6510e966526c903ff79771938f064f3f8caa34c2cbd6732589273a0496a7258f"
Nov 03 01:54:56.226916 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:56.226890    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh"
Nov 03 01:54:56.227031 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:56.226910    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl4vbh" event={"ID":"e4197e24-8ed5-453a-8273-6471221f3385","Type":"ContainerDied","Data":"d0345d0f30a38f6519854fbdf800c1f062679e88300342ef751227ca01d07dc3"}
Nov 03 01:54:56.227031 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:54:56.226946    5005 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0345d0f30a38f6519854fbdf800c1f062679e88300342ef751227ca01d07dc3"
Nov 03 01:55:00.712759 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:55:00.712722    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 01:55:07.359376 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359346    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-wjnv5"]
Nov 03 01:55:07.359846 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359684    5005 cpu_manager.go:414] "containerMap: removing container" podUID="ce68bd37-cfb9-44a5-8100-89db675d1265" containerName="pull"
Nov 03 01:55:07.359846 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359717    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce68bd37-cfb9-44a5-8100-89db675d1265" containerName="pull"
Nov 03 01:55:07.359846 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359740    5005 cpu_manager.go:414] "containerMap: removing container" podUID="69322313-2a08-49e8-9a58-6b06a89b79d4" containerName="pull"
Nov 03 01:55:07.359846 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359745    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69322313-2a08-49e8-9a58-6b06a89b79d4" containerName="pull"
Nov 03 01:55:07.359846 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359758    5005 cpu_manager.go:414] "containerMap: removing container" podUID="ce68bd37-cfb9-44a5-8100-89db675d1265" containerName="util"
Nov 03 01:55:07.359846 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359763    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce68bd37-cfb9-44a5-8100-89db675d1265" containerName="util"
Nov 03 01:55:07.359846 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359773    5005 cpu_manager.go:414] "containerMap: removing container" podUID="e4197e24-8ed5-453a-8273-6471221f3385" containerName="extract"
Nov 03 01:55:07.359846 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359778    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4197e24-8ed5-453a-8273-6471221f3385" containerName="extract"
Nov 03 01:55:07.359846 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359793    5005 cpu_manager.go:414] "containerMap: removing container" podUID="ce68bd37-cfb9-44a5-8100-89db675d1265" containerName="extract"
Nov 03 01:55:07.359846 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359798    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce68bd37-cfb9-44a5-8100-89db675d1265" containerName="extract"
Nov 03 01:55:07.359846 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359813    5005 cpu_manager.go:414] "containerMap: removing container" podUID="e4197e24-8ed5-453a-8273-6471221f3385" containerName="pull"
Nov 03 01:55:07.359846 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359818    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4197e24-8ed5-453a-8273-6471221f3385" containerName="pull"
Nov 03 01:55:07.359846 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359824    5005 cpu_manager.go:414] "containerMap: removing container" podUID="69322313-2a08-49e8-9a58-6b06a89b79d4" containerName="extract"
Nov 03 01:55:07.359846 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359828    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69322313-2a08-49e8-9a58-6b06a89b79d4" containerName="extract"
Nov 03 01:55:07.359846 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359836    5005 cpu_manager.go:414] "containerMap: removing container" podUID="e609f442-f2f5-49e4-99e1-05108a43d01f" containerName="extract"
Nov 03 01:55:07.359846 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359840    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e609f442-f2f5-49e4-99e1-05108a43d01f" containerName="extract"
Nov 03 01:55:07.360318 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359856    5005 cpu_manager.go:414] "containerMap: removing container" podUID="e609f442-f2f5-49e4-99e1-05108a43d01f" containerName="util"
Nov 03 01:55:07.360318 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359862    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e609f442-f2f5-49e4-99e1-05108a43d01f" containerName="util"
Nov 03 01:55:07.360318 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359867    5005 cpu_manager.go:414] "containerMap: removing container" podUID="e609f442-f2f5-49e4-99e1-05108a43d01f" containerName="pull"
Nov 03 01:55:07.360318 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359872    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e609f442-f2f5-49e4-99e1-05108a43d01f" containerName="pull"
Nov 03 01:55:07.360318 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359880    5005 cpu_manager.go:414] "containerMap: removing container" podUID="e4197e24-8ed5-453a-8273-6471221f3385" containerName="util"
Nov 03 01:55:07.360318 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359884    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4197e24-8ed5-453a-8273-6471221f3385" containerName="util"
Nov 03 01:55:07.360318 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359890    5005 cpu_manager.go:414] "containerMap: removing container" podUID="69322313-2a08-49e8-9a58-6b06a89b79d4" containerName="util"
Nov 03 01:55:07.360318 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359897    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69322313-2a08-49e8-9a58-6b06a89b79d4" containerName="util"
Nov 03 01:55:07.360318 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359946    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4197e24-8ed5-453a-8273-6471221f3385" containerName="extract"
Nov 03 01:55:07.360318 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359954    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce68bd37-cfb9-44a5-8100-89db675d1265" containerName="extract"
Nov 03 01:55:07.360318 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359960    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="69322313-2a08-49e8-9a58-6b06a89b79d4" containerName="extract"
Nov 03 01:55:07.360318 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.359965    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="e609f442-f2f5-49e4-99e1-05108a43d01f" containerName="extract"
Nov 03 01:55:07.364445 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.364426    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wjnv5"
Nov 03 01:55:07.461554 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.461523    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 01:55:07.461832 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.461801    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 01:55:07.461945 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.461898    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-gcxfl\""
Nov 03 01:55:07.463273 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.463252    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-wjnv5"]
Nov 03 01:55:07.468238 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.468221    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkqmt\" (UniqueName: \"kubernetes.io/projected/464b4a1a-cf6e-40d2-9859-3f10e7a00626-kube-api-access-nkqmt\") pod \"jobset-operator-674cd59dc5-wjnv5\" (UID: \"464b4a1a-cf6e-40d2-9859-3f10e7a00626\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wjnv5"
Nov 03 01:55:07.468297 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.468274    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/464b4a1a-cf6e-40d2-9859-3f10e7a00626-tmp\") pod \"jobset-operator-674cd59dc5-wjnv5\" (UID: \"464b4a1a-cf6e-40d2-9859-3f10e7a00626\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wjnv5"
Nov 03 01:55:07.569179 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.569148    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkqmt\" (UniqueName: \"kubernetes.io/projected/464b4a1a-cf6e-40d2-9859-3f10e7a00626-kube-api-access-nkqmt\") pod \"jobset-operator-674cd59dc5-wjnv5\" (UID: \"464b4a1a-cf6e-40d2-9859-3f10e7a00626\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wjnv5"
Nov 03 01:55:07.569320 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.569211    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/464b4a1a-cf6e-40d2-9859-3f10e7a00626-tmp\") pod \"jobset-operator-674cd59dc5-wjnv5\" (UID: \"464b4a1a-cf6e-40d2-9859-3f10e7a00626\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wjnv5"
Nov 03 01:55:07.569586 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.569569    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/464b4a1a-cf6e-40d2-9859-3f10e7a00626-tmp\") pod \"jobset-operator-674cd59dc5-wjnv5\" (UID: \"464b4a1a-cf6e-40d2-9859-3f10e7a00626\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wjnv5"
Nov 03 01:55:07.622624 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.622564    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkqmt\" (UniqueName: \"kubernetes.io/projected/464b4a1a-cf6e-40d2-9859-3f10e7a00626-kube-api-access-nkqmt\") pod \"jobset-operator-674cd59dc5-wjnv5\" (UID: \"464b4a1a-cf6e-40d2-9859-3f10e7a00626\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wjnv5"
Nov 03 01:55:07.685781 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:07.685743    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wjnv5"
Nov 03 01:55:08.048257 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:08.048223    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-wjnv5"]
Nov 03 01:55:08.055000 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:55:08.054971    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod464b4a1a_cf6e_40d2_9859_3f10e7a00626.slice/crio-30fa7d40e8555688db7bcb262c96dd374fe910baa13af10b254405b4a0892939 WatchSource:0}: Error finding container 30fa7d40e8555688db7bcb262c96dd374fe910baa13af10b254405b4a0892939: Status 404 returned error can't find the container with id 30fa7d40e8555688db7bcb262c96dd374fe910baa13af10b254405b4a0892939
Nov 03 01:55:08.280758 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:08.280727    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wjnv5" event={"ID":"464b4a1a-cf6e-40d2-9859-3f10e7a00626","Type":"ContainerStarted","Data":"30fa7d40e8555688db7bcb262c96dd374fe910baa13af10b254405b4a0892939"}
Nov 03 01:55:11.773507 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.773467    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"]
Nov 03 01:55:11.780934 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.780910    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"
Nov 03 01:55:11.806106 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.806082    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35514696-2346-49d3-9f0c-20f4e827c5fa-tmp\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-zbwkv\" (UID: \"35514696-2346-49d3-9f0c-20f4e827c5fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"
Nov 03 01:55:11.806247 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.806112    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35514696-2346-49d3-9f0c-20f4e827c5fa-webhook-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-zbwkv\" (UID: \"35514696-2346-49d3-9f0c-20f4e827c5fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"
Nov 03 01:55:11.806247 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.806148    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/35514696-2346-49d3-9f0c-20f4e827c5fa-manifests\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-zbwkv\" (UID: \"35514696-2346-49d3-9f0c-20f4e827c5fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"
Nov 03 01:55:11.806343 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.806311    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/35514696-2346-49d3-9f0c-20f4e827c5fa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-zbwkv\" (UID: \"35514696-2346-49d3-9f0c-20f4e827c5fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"
Nov 03 01:55:11.806395 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.806349    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/35514696-2346-49d3-9f0c-20f4e827c5fa-metrics-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-zbwkv\" (UID: \"35514696-2346-49d3-9f0c-20f4e827c5fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"
Nov 03 01:55:11.806444 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.806425    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5qkr\" (UniqueName: \"kubernetes.io/projected/35514696-2346-49d3-9f0c-20f4e827c5fa-kube-api-access-r5qkr\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-zbwkv\" (UID: \"35514696-2346-49d3-9f0c-20f4e827c5fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"
Nov 03 01:55:11.819757 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.819726    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"]
Nov 03 01:55:11.851649 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.851598    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 01:55:11.851859 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.851804    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 01:55:11.851859 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.851839    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 01:55:11.852336 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.852302    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hwpp2\""
Nov 03 01:55:11.859126 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.859106    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 01:55:11.859235 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.859143    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 01:55:11.907139 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.907110    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35514696-2346-49d3-9f0c-20f4e827c5fa-tmp\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-zbwkv\" (UID: \"35514696-2346-49d3-9f0c-20f4e827c5fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"
Nov 03 01:55:11.907139 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.907143    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35514696-2346-49d3-9f0c-20f4e827c5fa-webhook-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-zbwkv\" (UID: \"35514696-2346-49d3-9f0c-20f4e827c5fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"
Nov 03 01:55:11.907578 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.907175    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/35514696-2346-49d3-9f0c-20f4e827c5fa-manifests\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-zbwkv\" (UID: \"35514696-2346-49d3-9f0c-20f4e827c5fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"
Nov 03 01:55:11.907578 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.907202    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/35514696-2346-49d3-9f0c-20f4e827c5fa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-zbwkv\" (UID: \"35514696-2346-49d3-9f0c-20f4e827c5fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"
Nov 03 01:55:11.907578 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.907218    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/35514696-2346-49d3-9f0c-20f4e827c5fa-metrics-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-zbwkv\" (UID: \"35514696-2346-49d3-9f0c-20f4e827c5fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"
Nov 03 01:55:11.907578 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.907247    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5qkr\" (UniqueName: \"kubernetes.io/projected/35514696-2346-49d3-9f0c-20f4e827c5fa-kube-api-access-r5qkr\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-zbwkv\" (UID: \"35514696-2346-49d3-9f0c-20f4e827c5fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"
Nov 03 01:55:11.910073 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.910049    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/35514696-2346-49d3-9f0c-20f4e827c5fa-manifests\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-zbwkv\" (UID: \"35514696-2346-49d3-9f0c-20f4e827c5fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"
Nov 03 01:55:11.910450 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.910424    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35514696-2346-49d3-9f0c-20f4e827c5fa-tmp\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-zbwkv\" (UID: \"35514696-2346-49d3-9f0c-20f4e827c5fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"
Nov 03 01:55:11.913151 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.913108    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/35514696-2346-49d3-9f0c-20f4e827c5fa-metrics-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-zbwkv\" (UID: \"35514696-2346-49d3-9f0c-20f4e827c5fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"
Nov 03 01:55:11.914354 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.914333    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/35514696-2346-49d3-9f0c-20f4e827c5fa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-zbwkv\" (UID: \"35514696-2346-49d3-9f0c-20f4e827c5fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"
Nov 03 01:55:11.914526 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.914506    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35514696-2346-49d3-9f0c-20f4e827c5fa-webhook-cert\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-zbwkv\" (UID: \"35514696-2346-49d3-9f0c-20f4e827c5fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"
Nov 03 01:55:11.962239 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:11.962209    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5qkr\" (UniqueName: \"kubernetes.io/projected/35514696-2346-49d3-9f0c-20f4e827c5fa-kube-api-access-r5qkr\") pod \"opendatahub-operator-controller-manager-864bf9f5c8-zbwkv\" (UID: \"35514696-2346-49d3-9f0c-20f4e827c5fa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"
Nov 03 01:55:12.100318 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:12.100243    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"
Nov 03 01:55:12.643826 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:12.643789    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"]
Nov 03 01:55:12.681044 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:55:12.681014    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35514696_2346_49d3_9f0c_20f4e827c5fa.slice/crio-1e9cad5a3ca5fefe479e1cf16bdd4b22647c60079f7de736ace1a70868a9b6cc WatchSource:0}: Error finding container 1e9cad5a3ca5fefe479e1cf16bdd4b22647c60079f7de736ace1a70868a9b6cc: Status 404 returned error can't find the container with id 1e9cad5a3ca5fefe479e1cf16bdd4b22647c60079f7de736ace1a70868a9b6cc
Nov 03 01:55:13.313102 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:13.313062    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wjnv5" event={"ID":"464b4a1a-cf6e-40d2-9859-3f10e7a00626","Type":"ContainerStarted","Data":"76ba6dad4532c0b8b48103efc8db2f224bc90067cae41907f1584a564b74a739"}
Nov 03 01:55:13.314318 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:13.314294    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv" event={"ID":"35514696-2346-49d3-9f0c-20f4e827c5fa","Type":"ContainerStarted","Data":"1e9cad5a3ca5fefe479e1cf16bdd4b22647c60079f7de736ace1a70868a9b6cc"}
Nov 03 01:55:13.422220 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:13.422173    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wjnv5" podStartSLOduration=1.570613096 podStartE2EDuration="6.42216052s" totalImagesPullingTime="4.851547424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:13.420333355 +0000 UTC m=+240.434574723" watchObservedRunningTime="2026-11-03 01:55:13.42216052 +0000 UTC m=+240.436401886"
Nov 03 01:55:18.339677 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:18.339638    5005 generic.go:356] "Generic (PLEG): container finished" podID="35514696-2346-49d3-9f0c-20f4e827c5fa" containerID="5c9d470e8825afb9236f91b1411e47224c0dd18fc6aa00ca2aa0c19907ca31dd" exitCode=0
Nov 03 01:55:18.340073 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:18.339728    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv" event={"ID":"35514696-2346-49d3-9f0c-20f4e827c5fa","Type":"ContainerDied","Data":"5c9d470e8825afb9236f91b1411e47224c0dd18fc6aa00ca2aa0c19907ca31dd"}
Nov 03 01:55:19.346170 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:19.346128    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv" event={"ID":"35514696-2346-49d3-9f0c-20f4e827c5fa","Type":"ContainerStarted","Data":"c852e46cdbd8b383b8f0159af707368df91f314e345702bff21e0c58108d66e0"}
Nov 03 01:55:19.346606 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:19.346278    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"
Nov 03 01:55:19.466934 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:19.466849    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv" podStartSLOduration=2.4408753389999998 podStartE2EDuration="8.466830366s" totalImagesPullingTime="5.597269869s" totalInitContainerRuntime="428.685158ms" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:19.466136612 +0000 UTC m=+246.480377978" watchObservedRunningTime="2026-11-03 01:55:19.466830366 +0000 UTC m=+246.481071732"
Nov 03 01:55:23.727259 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:23.727223    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7f45k"]
Nov 03 01:55:23.730515 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:23.730498    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7f45k"
Nov 03 01:55:23.820894 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:23.820855    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59b02e0c-5c2e-4370-8c32-9b43d4cf727f-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7f45k\" (UID: \"59b02e0c-5c2e-4370-8c32-9b43d4cf727f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7f45k"
Nov 03 01:55:23.820894 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:23.820895    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cmp4\" (UniqueName: \"kubernetes.io/projected/59b02e0c-5c2e-4370-8c32-9b43d4cf727f-kube-api-access-8cmp4\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7f45k\" (UID: \"59b02e0c-5c2e-4370-8c32-9b43d4cf727f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7f45k"
Nov 03 01:55:23.835543 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:23.835518    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-7ttgw\""
Nov 03 01:55:23.835675 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:23.835554    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 03 01:55:23.835675 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:23.835584    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 03 01:55:23.837306 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:23.837287    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7f45k"]
Nov 03 01:55:23.922332 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:23.922297    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59b02e0c-5c2e-4370-8c32-9b43d4cf727f-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7f45k\" (UID: \"59b02e0c-5c2e-4370-8c32-9b43d4cf727f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7f45k"
Nov 03 01:55:23.922499 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:23.922337    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cmp4\" (UniqueName: \"kubernetes.io/projected/59b02e0c-5c2e-4370-8c32-9b43d4cf727f-kube-api-access-8cmp4\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7f45k\" (UID: \"59b02e0c-5c2e-4370-8c32-9b43d4cf727f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7f45k"
Nov 03 01:55:23.922727 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:23.922684    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59b02e0c-5c2e-4370-8c32-9b43d4cf727f-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7f45k\" (UID: \"59b02e0c-5c2e-4370-8c32-9b43d4cf727f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7f45k"
Nov 03 01:55:23.979050 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:23.978987    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cmp4\" (UniqueName: \"kubernetes.io/projected/59b02e0c-5c2e-4370-8c32-9b43d4cf727f-kube-api-access-8cmp4\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7f45k\" (UID: \"59b02e0c-5c2e-4370-8c32-9b43d4cf727f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7f45k"
Nov 03 01:55:24.042099 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:24.042066    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7f45k"
Nov 03 01:55:24.466030 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:24.465996    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7f45k"]
Nov 03 01:55:24.474460 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:55:24.474431    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b02e0c_5c2e_4370_8c32_9b43d4cf727f.slice/crio-b48b8a57363eb44c68af98a6fab01f4511fd2b9e653958637e047eca44694707 WatchSource:0}: Error finding container b48b8a57363eb44c68af98a6fab01f4511fd2b9e653958637e047eca44694707: Status 404 returned error can't find the container with id b48b8a57363eb44c68af98a6fab01f4511fd2b9e653958637e047eca44694707
Nov 03 01:55:25.374351 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:25.374316    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7f45k" event={"ID":"59b02e0c-5c2e-4370-8c32-9b43d4cf727f","Type":"ContainerStarted","Data":"b48b8a57363eb44c68af98a6fab01f4511fd2b9e653958637e047eca44694707"}
Nov 03 01:55:29.398044 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:29.398005    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7f45k" event={"ID":"59b02e0c-5c2e-4370-8c32-9b43d4cf727f","Type":"ContainerStarted","Data":"3ae0378c9e84a06f5071eb64c7f75301205c74df4e774142443850e39cdd1541"}
Nov 03 01:55:29.600758 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:29.600706    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7f45k" podStartSLOduration=2.565828358 podStartE2EDuration="6.600675702s" totalImagesPullingTime="4.034847344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:29.599183153 +0000 UTC m=+256.613424520" watchObservedRunningTime="2026-11-03 01:55:29.600675702 +0000 UTC m=+256.614917067"
Nov 03 01:55:30.353680 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:30.353646    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864bf9f5c8-zbwkv"
Nov 03 01:55:30.712617 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:55:30.712542    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 01:55:30.871873 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:30.871840    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-sxxkw"]
Nov 03 01:55:30.875318 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:30.875297    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sxxkw"
Nov 03 01:55:30.928009 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:30.927971    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 03 01:55:30.928009 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:30.927988    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 01:55:30.928206 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:30.928023    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 01:55:30.928206 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:30.928148    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bck82\""
Nov 03 01:55:30.928206 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:30.928200    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 03 01:55:30.930472 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:30.930451    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-sxxkw"]
Nov 03 01:55:30.978419 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:30.978357    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/75f9fa81-69b8-4cfe-acdb-8e68d6e93d02-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-sxxkw\" (UID: \"75f9fa81-69b8-4cfe-acdb-8e68d6e93d02\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sxxkw"
Nov 03 01:55:30.978419 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:30.978384    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p5gj\" (UniqueName: \"kubernetes.io/projected/75f9fa81-69b8-4cfe-acdb-8e68d6e93d02-kube-api-access-5p5gj\") pod \"kuadrant-console-plugin-66c4fb575d-sxxkw\" (UID: \"75f9fa81-69b8-4cfe-acdb-8e68d6e93d02\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sxxkw"
Nov 03 01:55:30.978599 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:30.978420    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/75f9fa81-69b8-4cfe-acdb-8e68d6e93d02-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-sxxkw\" (UID: \"75f9fa81-69b8-4cfe-acdb-8e68d6e93d02\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sxxkw"
Nov 03 01:55:31.079840 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:31.079810    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/75f9fa81-69b8-4cfe-acdb-8e68d6e93d02-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-sxxkw\" (UID: \"75f9fa81-69b8-4cfe-acdb-8e68d6e93d02\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sxxkw"
Nov 03 01:55:31.079840 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:31.079839    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p5gj\" (UniqueName: \"kubernetes.io/projected/75f9fa81-69b8-4cfe-acdb-8e68d6e93d02-kube-api-access-5p5gj\") pod \"kuadrant-console-plugin-66c4fb575d-sxxkw\" (UID: \"75f9fa81-69b8-4cfe-acdb-8e68d6e93d02\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sxxkw"
Nov 03 01:55:31.080021 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:31.079875    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/75f9fa81-69b8-4cfe-acdb-8e68d6e93d02-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-sxxkw\" (UID: \"75f9fa81-69b8-4cfe-acdb-8e68d6e93d02\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sxxkw"
Nov 03 01:55:31.080534 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:31.080502    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/75f9fa81-69b8-4cfe-acdb-8e68d6e93d02-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-sxxkw\" (UID: \"75f9fa81-69b8-4cfe-acdb-8e68d6e93d02\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sxxkw"
Nov 03 01:55:31.082233 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:31.082211    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/75f9fa81-69b8-4cfe-acdb-8e68d6e93d02-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-sxxkw\" (UID: \"75f9fa81-69b8-4cfe-acdb-8e68d6e93d02\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sxxkw"
Nov 03 01:55:31.140973 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:31.140950    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p5gj\" (UniqueName: \"kubernetes.io/projected/75f9fa81-69b8-4cfe-acdb-8e68d6e93d02-kube-api-access-5p5gj\") pod \"kuadrant-console-plugin-66c4fb575d-sxxkw\" (UID: \"75f9fa81-69b8-4cfe-acdb-8e68d6e93d02\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sxxkw"
Nov 03 01:55:31.187815 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:31.187793    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sxxkw"
Nov 03 01:55:31.562566 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:31.562531    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-sxxkw"]
Nov 03 01:55:31.569479 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:55:31.569437    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75f9fa81_69b8_4cfe_acdb_8e68d6e93d02.slice/crio-c12db7db5cfbeab1569f4a91b19c23393f972d08036a066c92843d1619d18429 WatchSource:0}: Error finding container c12db7db5cfbeab1569f4a91b19c23393f972d08036a066c92843d1619d18429: Status 404 returned error can't find the container with id c12db7db5cfbeab1569f4a91b19c23393f972d08036a066c92843d1619d18429
Nov 03 01:55:32.413976 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:32.413941    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sxxkw" event={"ID":"75f9fa81-69b8-4cfe-acdb-8e68d6e93d02","Type":"ContainerStarted","Data":"c12db7db5cfbeab1569f4a91b19c23393f972d08036a066c92843d1619d18429"}
Nov 03 01:55:32.458704 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:32.458662    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-7mght"]
Nov 03 01:55:32.463270 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:32.463255    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7mght"
Nov 03 01:55:32.514655 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:32.514630    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 01:55:32.516366 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:32.516341    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-7mght"]
Nov 03 01:55:32.560745 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:32.560718    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-4hnkm\""
Nov 03 01:55:32.560863 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:32.560775    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 01:55:32.592787 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:32.592761    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5szd\" (UniqueName: \"kubernetes.io/projected/f41c2c39-e69a-4811-ae75-e5869f793406-kube-api-access-k5szd\") pod \"cert-manager-webhook-5dbbbf75cc-7mght\" (UID: \"f41c2c39-e69a-4811-ae75-e5869f793406\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7mght"
Nov 03 01:55:32.592896 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:32.592817    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f41c2c39-e69a-4811-ae75-e5869f793406-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-7mght\" (UID: \"f41c2c39-e69a-4811-ae75-e5869f793406\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7mght"
Nov 03 01:55:32.693413 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:32.693345    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5szd\" (UniqueName: \"kubernetes.io/projected/f41c2c39-e69a-4811-ae75-e5869f793406-kube-api-access-k5szd\") pod \"cert-manager-webhook-5dbbbf75cc-7mght\" (UID: \"f41c2c39-e69a-4811-ae75-e5869f793406\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7mght"
Nov 03 01:55:32.693413 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:32.693393    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f41c2c39-e69a-4811-ae75-e5869f793406-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-7mght\" (UID: \"f41c2c39-e69a-4811-ae75-e5869f793406\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7mght"
Nov 03 01:55:32.767174 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:32.767143    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f41c2c39-e69a-4811-ae75-e5869f793406-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-7mght\" (UID: \"f41c2c39-e69a-4811-ae75-e5869f793406\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7mght"
Nov 03 01:55:32.767321 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:32.767301    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5szd\" (UniqueName: \"kubernetes.io/projected/f41c2c39-e69a-4811-ae75-e5869f793406-kube-api-access-k5szd\") pod \"cert-manager-webhook-5dbbbf75cc-7mght\" (UID: \"f41c2c39-e69a-4811-ae75-e5869f793406\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7mght"
Nov 03 01:55:32.775084 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:32.775063    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7mght"
Nov 03 01:55:33.151043 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:33.148188    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-7mght"]
Nov 03 01:55:33.154710 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:55:33.154671    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf41c2c39_e69a_4811_ae75_e5869f793406.slice/crio-01d76adf64edb87b35b06bb022fa06378eb32f846d025625b2f06197ce9dec58 WatchSource:0}: Error finding container 01d76adf64edb87b35b06bb022fa06378eb32f846d025625b2f06197ce9dec58: Status 404 returned error can't find the container with id 01d76adf64edb87b35b06bb022fa06378eb32f846d025625b2f06197ce9dec58
Nov 03 01:55:33.421210 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:33.421137    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7mght" event={"ID":"f41c2c39-e69a-4811-ae75-e5869f793406","Type":"ContainerStarted","Data":"01d76adf64edb87b35b06bb022fa06378eb32f846d025625b2f06197ce9dec58"}
Nov 03 01:55:39.457364 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:39.457324    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sxxkw" event={"ID":"75f9fa81-69b8-4cfe-acdb-8e68d6e93d02","Type":"ContainerStarted","Data":"fc1689afdd83b4d6ec380273e1a790d876066ccc5e347fd7d3b4407f202c750b"}
Nov 03 01:55:39.459014 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:39.458987    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7mght" event={"ID":"f41c2c39-e69a-4811-ae75-e5869f793406","Type":"ContainerStarted","Data":"c60968b67081baa55bf609c01454831343fa1a666ec7057ce19918005bb8fdeb"}
Nov 03 01:55:39.459135 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:39.459049    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7mght"
Nov 03 01:55:39.562799 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:39.562757    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-sxxkw" podStartSLOduration=2.156183744 podStartE2EDuration="9.562742221s" totalImagesPullingTime="7.406558477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:39.562132343 +0000 UTC m=+266.576373708" watchObservedRunningTime="2026-11-03 01:55:39.562742221 +0000 UTC m=+266.576983586"
Nov 03 01:55:39.668237 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:39.668184    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7mght" podStartSLOduration=2.325558649 podStartE2EDuration="7.668168832s" totalImagesPullingTime="5.342610183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:39.66756327 +0000 UTC m=+266.681804636" watchObservedRunningTime="2026-11-03 01:55:39.668168832 +0000 UTC m=+266.682410196"
Nov 03 01:55:44.469572 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:44.469535    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-87gr8"]
Nov 03 01:55:44.559583 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:44.559547    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-87gr8"]
Nov 03 01:55:44.559749 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:44.559643    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-87gr8"
Nov 03 01:55:44.608064 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:44.608036    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 01:55:44.700647 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:44.700617    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4pks\" (UniqueName: \"kubernetes.io/projected/201f8059-9119-474f-b3b7-77f1595da7a7-kube-api-access-q4pks\") pod \"limitador-limitador-74dfcfb894-87gr8\" (UID: \"201f8059-9119-474f-b3b7-77f1595da7a7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-87gr8"
Nov 03 01:55:44.700854 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:44.700763    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/201f8059-9119-474f-b3b7-77f1595da7a7-config-file\") pod \"limitador-limitador-74dfcfb894-87gr8\" (UID: \"201f8059-9119-474f-b3b7-77f1595da7a7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-87gr8"
Nov 03 01:55:44.707895 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:44.707864    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-87gr8"]
Nov 03 01:55:44.801568 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:44.801536    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4pks\" (UniqueName: \"kubernetes.io/projected/201f8059-9119-474f-b3b7-77f1595da7a7-kube-api-access-q4pks\") pod \"limitador-limitador-74dfcfb894-87gr8\" (UID: \"201f8059-9119-474f-b3b7-77f1595da7a7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-87gr8"
Nov 03 01:55:44.801758 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:44.801616    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/201f8059-9119-474f-b3b7-77f1595da7a7-config-file\") pod \"limitador-limitador-74dfcfb894-87gr8\" (UID: \"201f8059-9119-474f-b3b7-77f1595da7a7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-87gr8"
Nov 03 01:55:44.802279 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:44.802253    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/201f8059-9119-474f-b3b7-77f1595da7a7-config-file\") pod \"limitador-limitador-74dfcfb894-87gr8\" (UID: \"201f8059-9119-474f-b3b7-77f1595da7a7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-87gr8"
Nov 03 01:55:44.854507 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:44.854477    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4pks\" (UniqueName: \"kubernetes.io/projected/201f8059-9119-474f-b3b7-77f1595da7a7-kube-api-access-q4pks\") pod \"limitador-limitador-74dfcfb894-87gr8\" (UID: \"201f8059-9119-474f-b3b7-77f1595da7a7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-87gr8"
Nov 03 01:55:44.882992 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:44.882963    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-87gr8"
Nov 03 01:55:44.937821 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:44.937800    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6dc8f585b4-g6dbz"]
Nov 03 01:55:44.971336 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:44.971303    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-g6dbz"
Nov 03 01:55:44.994816 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:44.994785    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dc8f585b4-g6dbz"]
Nov 03 01:55:45.026042 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:45.026013    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 01:55:45.026163 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:45.026013    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-l7n64\""
Nov 03 01:55:45.026163 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:45.026050    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 01:55:45.104496 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:45.104416    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b1e1c0-194f-47a1-8a7b-7ab2ff3dbf2d-tls-certs\") pod \"kube-auth-proxy-6dc8f585b4-g6dbz\" (UID: \"d3b1e1c0-194f-47a1-8a7b-7ab2ff3dbf2d\") " pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-g6dbz"
Nov 03 01:55:45.104649 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:45.104523    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67bjz\" (UniqueName: \"kubernetes.io/projected/d3b1e1c0-194f-47a1-8a7b-7ab2ff3dbf2d-kube-api-access-67bjz\") pod \"kube-auth-proxy-6dc8f585b4-g6dbz\" (UID: \"d3b1e1c0-194f-47a1-8a7b-7ab2ff3dbf2d\") " pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-g6dbz"
Nov 03 01:55:45.104649 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:45.104566    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3b1e1c0-194f-47a1-8a7b-7ab2ff3dbf2d-tmp\") pod \"kube-auth-proxy-6dc8f585b4-g6dbz\" (UID: \"d3b1e1c0-194f-47a1-8a7b-7ab2ff3dbf2d\") " pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-g6dbz"
Nov 03 01:55:45.205773 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:45.205739    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67bjz\" (UniqueName: \"kubernetes.io/projected/d3b1e1c0-194f-47a1-8a7b-7ab2ff3dbf2d-kube-api-access-67bjz\") pod \"kube-auth-proxy-6dc8f585b4-g6dbz\" (UID: \"d3b1e1c0-194f-47a1-8a7b-7ab2ff3dbf2d\") " pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-g6dbz"
Nov 03 01:55:45.205947 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:45.205790    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3b1e1c0-194f-47a1-8a7b-7ab2ff3dbf2d-tmp\") pod \"kube-auth-proxy-6dc8f585b4-g6dbz\" (UID: \"d3b1e1c0-194f-47a1-8a7b-7ab2ff3dbf2d\") " pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-g6dbz"
Nov 03 01:55:45.205947 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:45.205839    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b1e1c0-194f-47a1-8a7b-7ab2ff3dbf2d-tls-certs\") pod \"kube-auth-proxy-6dc8f585b4-g6dbz\" (UID: \"d3b1e1c0-194f-47a1-8a7b-7ab2ff3dbf2d\") " pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-g6dbz"
Nov 03 01:55:45.208026 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:45.207998    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3b1e1c0-194f-47a1-8a7b-7ab2ff3dbf2d-tmp\") pod \"kube-auth-proxy-6dc8f585b4-g6dbz\" (UID: \"d3b1e1c0-194f-47a1-8a7b-7ab2ff3dbf2d\") " pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-g6dbz"
Nov 03 01:55:45.208270 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:45.208252    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b1e1c0-194f-47a1-8a7b-7ab2ff3dbf2d-tls-certs\") pod \"kube-auth-proxy-6dc8f585b4-g6dbz\" (UID: \"d3b1e1c0-194f-47a1-8a7b-7ab2ff3dbf2d\") " pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-g6dbz"
Nov 03 01:55:45.258576 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:45.258552    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67bjz\" (UniqueName: \"kubernetes.io/projected/d3b1e1c0-194f-47a1-8a7b-7ab2ff3dbf2d-kube-api-access-67bjz\") pod \"kube-auth-proxy-6dc8f585b4-g6dbz\" (UID: \"d3b1e1c0-194f-47a1-8a7b-7ab2ff3dbf2d\") " pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-g6dbz"
Nov 03 01:55:45.299389 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:45.299362    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-87gr8"]
Nov 03 01:55:45.306023 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:55:45.305998    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201f8059_9119_474f_b3b7_77f1595da7a7.slice/crio-ee0ff12634a44275ce9dad16b8530c947e4e53a6980b15ad82ebc84ed2eb2ff7 WatchSource:0}: Error finding container ee0ff12634a44275ce9dad16b8530c947e4e53a6980b15ad82ebc84ed2eb2ff7: Status 404 returned error can't find the container with id ee0ff12634a44275ce9dad16b8530c947e4e53a6980b15ad82ebc84ed2eb2ff7
Nov 03 01:55:45.307880 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:45.307863    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-g6dbz"
Nov 03 01:55:45.465251 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:45.465225    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-7mght"
Nov 03 01:55:45.488155 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:45.488110    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-87gr8" event={"ID":"201f8059-9119-474f-b3b7-77f1595da7a7","Type":"ContainerStarted","Data":"ee0ff12634a44275ce9dad16b8530c947e4e53a6980b15ad82ebc84ed2eb2ff7"}
Nov 03 01:55:45.720804 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:45.720725    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dc8f585b4-g6dbz"]
Nov 03 01:55:45.724579 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:55:45.724542    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b1e1c0_194f_47a1_8a7b_7ab2ff3dbf2d.slice/crio-8b1542976498f2930d1be7e1af5c84e5429285992b965630bfea4396d89c3b68 WatchSource:0}: Error finding container 8b1542976498f2930d1be7e1af5c84e5429285992b965630bfea4396d89c3b68: Status 404 returned error can't find the container with id 8b1542976498f2930d1be7e1af5c84e5429285992b965630bfea4396d89c3b68
Nov 03 01:55:46.493592 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:46.493557    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-g6dbz" event={"ID":"d3b1e1c0-194f-47a1-8a7b-7ab2ff3dbf2d","Type":"ContainerStarted","Data":"8b1542976498f2930d1be7e1af5c84e5429285992b965630bfea4396d89c3b68"}
Nov 03 01:55:49.513942 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:49.513891    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-87gr8" event={"ID":"201f8059-9119-474f-b3b7-77f1595da7a7","Type":"ContainerStarted","Data":"b2df2ee814593904294dc0e30e8c31fe9f5a73552b4e5f56862c438c2b78f9fd"}
Nov 03 01:55:49.514411 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:49.513972    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-87gr8"
Nov 03 01:55:49.664476 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:49.664373    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-87gr8" podStartSLOduration=1.559486028 podStartE2EDuration="5.664355877s" totalImagesPullingTime="4.104869849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:49.663027622 +0000 UTC m=+276.677268989" watchObservedRunningTime="2026-11-03 01:55:49.664355877 +0000 UTC m=+276.678597241"
Nov 03 01:55:50.523438 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:50.523398    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-g6dbz" event={"ID":"d3b1e1c0-194f-47a1-8a7b-7ab2ff3dbf2d","Type":"ContainerStarted","Data":"0dbdb612cdb109b2756cd5ce28a0ff44dea0309342308c76b11f738efe934720"}
Nov 03 01:55:50.631009 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:55:50.630960    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6dc8f585b4-g6dbz" podStartSLOduration=2.560512428 podStartE2EDuration="6.630946409s" totalImagesPullingTime="4.070433981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:50.629513374 +0000 UTC m=+277.643754743" watchObservedRunningTime="2026-11-03 01:55:50.630946409 +0000 UTC m=+277.645187774"
Nov 03 01:56:00.524557 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:00.524529    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-87gr8"
Nov 03 01:56:00.711583 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:56:00.711550    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 01:56:04.916653 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:04.916615    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-87gr8"]
Nov 03 01:56:04.917151 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:04.916901    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-87gr8" podUID="201f8059-9119-474f-b3b7-77f1595da7a7" containerName="limitador" containerID="cri-o://b2df2ee814593904294dc0e30e8c31fe9f5a73552b4e5f56862c438c2b78f9fd" gracePeriod=30
Nov 03 01:56:05.592429 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:05.592398    5005 generic.go:356] "Generic (PLEG): container finished" podID="201f8059-9119-474f-b3b7-77f1595da7a7" containerID="b2df2ee814593904294dc0e30e8c31fe9f5a73552b4e5f56862c438c2b78f9fd" exitCode=0
Nov 03 01:56:05.592574 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:05.592472    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-87gr8" event={"ID":"201f8059-9119-474f-b3b7-77f1595da7a7","Type":"ContainerDied","Data":"b2df2ee814593904294dc0e30e8c31fe9f5a73552b4e5f56862c438c2b78f9fd"}
Nov 03 01:56:05.860167 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:05.860143    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-87gr8"
Nov 03 01:56:05.864235 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:05.864214    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/201f8059-9119-474f-b3b7-77f1595da7a7-config-file\" (UniqueName: \"kubernetes.io/configmap/201f8059-9119-474f-b3b7-77f1595da7a7-config-file\") pod \"201f8059-9119-474f-b3b7-77f1595da7a7\" (UID: \"201f8059-9119-474f-b3b7-77f1595da7a7\") "
Nov 03 01:56:05.864350 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:05.864297    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/201f8059-9119-474f-b3b7-77f1595da7a7-kube-api-access-q4pks\" (UniqueName: \"kubernetes.io/projected/201f8059-9119-474f-b3b7-77f1595da7a7-kube-api-access-q4pks\") pod \"201f8059-9119-474f-b3b7-77f1595da7a7\" (UID: \"201f8059-9119-474f-b3b7-77f1595da7a7\") "
Nov 03 01:56:05.864520 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:05.864500    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201f8059-9119-474f-b3b7-77f1595da7a7-config-file" pod "201f8059-9119-474f-b3b7-77f1595da7a7" (UID: "201f8059-9119-474f-b3b7-77f1595da7a7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:56:05.866187 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:05.866167    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201f8059-9119-474f-b3b7-77f1595da7a7-kube-api-access-q4pks" pod "201f8059-9119-474f-b3b7-77f1595da7a7" (UID: "201f8059-9119-474f-b3b7-77f1595da7a7"). InnerVolumeSpecName "kube-api-access-q4pks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:56:05.965094 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:05.965065    5005 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/201f8059-9119-474f-b3b7-77f1595da7a7-config-file\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:56:05.965094 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:05.965091    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4pks\" (UniqueName: \"kubernetes.io/projected/201f8059-9119-474f-b3b7-77f1595da7a7-kube-api-access-q4pks\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:56:06.598508 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:06.598470    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-87gr8" event={"ID":"201f8059-9119-474f-b3b7-77f1595da7a7","Type":"ContainerDied","Data":"ee0ff12634a44275ce9dad16b8530c947e4e53a6980b15ad82ebc84ed2eb2ff7"}
Nov 03 01:56:06.598508 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:06.598506    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-87gr8"
Nov 03 01:56:06.598759 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:06.598545    5005 scope.go:122] "RemoveContainer" containerID="b2df2ee814593904294dc0e30e8c31fe9f5a73552b4e5f56862c438c2b78f9fd"
Nov 03 01:56:06.750822 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:06.750786    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-87gr8"]
Nov 03 01:56:06.797286 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:06.797176    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-87gr8"]
Nov 03 01:56:07.107154 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:07.107121    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="201f8059-9119-474f-b3b7-77f1595da7a7" path="/var/lib/kubelet/pods/201f8059-9119-474f-b3b7-77f1595da7a7/volumes"
Nov 03 01:56:13.121090 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:13.121068    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 01:56:13.121489 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:13.121434    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 01:56:26.291916 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:26.291843    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-nv5dz"]
Nov 03 01:56:26.292307 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:26.292206    5005 cpu_manager.go:414] "containerMap: removing container" podUID="201f8059-9119-474f-b3b7-77f1595da7a7" containerName="limitador"
Nov 03 01:56:26.292307 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:26.292217    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="201f8059-9119-474f-b3b7-77f1595da7a7" containerName="limitador"
Nov 03 01:56:26.292307 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:26.292273    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="201f8059-9119-474f-b3b7-77f1595da7a7" containerName="limitador"
Nov 03 01:56:26.296436 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:26.296419    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nv5dz"
Nov 03 01:56:26.330783 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:26.330757    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4fabfb92-7556-4a30-a90f-8419b2b2ad95-data\") pod \"postgres-7cdbf6dc44-nv5dz\" (UID: \"4fabfb92-7556-4a30-a90f-8419b2b2ad95\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nv5dz"
Nov 03 01:56:26.330939 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:26.330830    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp6rr\" (UniqueName: \"kubernetes.io/projected/4fabfb92-7556-4a30-a90f-8419b2b2ad95-kube-api-access-qp6rr\") pod \"postgres-7cdbf6dc44-nv5dz\" (UID: \"4fabfb92-7556-4a30-a90f-8419b2b2ad95\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nv5dz"
Nov 03 01:56:26.346812 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:26.346786    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:56:26.348632 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:26.348608    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-nv5dz"]
Nov 03 01:56:26.358057 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:26.358035    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 03 01:56:26.358184 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:26.358078    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:56:26.393057 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:26.393039    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-gc8d7\""
Nov 03 01:56:26.431789 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:26.431767    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4fabfb92-7556-4a30-a90f-8419b2b2ad95-data\") pod \"postgres-7cdbf6dc44-nv5dz\" (UID: \"4fabfb92-7556-4a30-a90f-8419b2b2ad95\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nv5dz"
Nov 03 01:56:26.431895 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:26.431826    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp6rr\" (UniqueName: \"kubernetes.io/projected/4fabfb92-7556-4a30-a90f-8419b2b2ad95-kube-api-access-qp6rr\") pod \"postgres-7cdbf6dc44-nv5dz\" (UID: \"4fabfb92-7556-4a30-a90f-8419b2b2ad95\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nv5dz"
Nov 03 01:56:26.432142 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:26.432125    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4fabfb92-7556-4a30-a90f-8419b2b2ad95-data\") pod \"postgres-7cdbf6dc44-nv5dz\" (UID: \"4fabfb92-7556-4a30-a90f-8419b2b2ad95\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nv5dz"
Nov 03 01:56:26.484909 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:26.484879    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp6rr\" (UniqueName: \"kubernetes.io/projected/4fabfb92-7556-4a30-a90f-8419b2b2ad95-kube-api-access-qp6rr\") pod \"postgres-7cdbf6dc44-nv5dz\" (UID: \"4fabfb92-7556-4a30-a90f-8419b2b2ad95\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nv5dz"
Nov 03 01:56:26.608639 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:26.608545    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nv5dz"
Nov 03 01:56:26.973774 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:26.973681    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-nv5dz"]
Nov 03 01:56:26.980839 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:56:26.980815    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fabfb92_7556_4a30_a90f_8419b2b2ad95.slice/crio-da235ef521fa0ac8a5ccfa460adef8acaf710f67e1c45a387455c6452f4f027b WatchSource:0}: Error finding container da235ef521fa0ac8a5ccfa460adef8acaf710f67e1c45a387455c6452f4f027b: Status 404 returned error can't find the container with id da235ef521fa0ac8a5ccfa460adef8acaf710f67e1c45a387455c6452f4f027b
Nov 03 01:56:26.982447 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:26.982428    5005 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:56:27.698550 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:27.698517    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nv5dz" event={"ID":"4fabfb92-7556-4a30-a90f-8419b2b2ad95","Type":"ContainerStarted","Data":"da235ef521fa0ac8a5ccfa460adef8acaf710f67e1c45a387455c6452f4f027b"}
Nov 03 01:56:30.722472 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:56:30.722432    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 01:56:35.746754 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:35.746657    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nv5dz" event={"ID":"4fabfb92-7556-4a30-a90f-8419b2b2ad95","Type":"ContainerStarted","Data":"d3f75a7c1ed1a7dc5e2e6ebb1bdea366cdfe2000b0b1241f575e30400d8018db"}
Nov 03 01:56:35.747224 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:35.746864    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nv5dz"
Nov 03 01:56:35.852300 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:35.852225    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nv5dz" podStartSLOduration=1.419045151 podStartE2EDuration="9.852209206s" totalImagesPullingTime="8.433164055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:56:35.850912036 +0000 UTC m=+322.865153401" watchObservedRunningTime="2026-11-03 01:56:35.852209206 +0000 UTC m=+322.866450575"
Nov 03 01:56:41.780607 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:41.780574    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nv5dz"
Nov 03 01:56:49.514051 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:49.514015    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58fdd44778-vw5qm"]
Nov 03 01:56:49.517855 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:49.517839    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58fdd44778-vw5qm"
Nov 03 01:56:49.567072 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:49.567046    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 01:56:49.620889 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:49.620761    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 01:56:49.620889 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:49.620793    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 01:56:49.620889 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:49.620848    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 01:56:49.620889 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:49.620877    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-whxgk\""
Nov 03 01:56:49.621158 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:49.620925    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 01:56:49.622672 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:49.622650    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58fdd44778-vw5qm"]
Nov 03 01:56:49.634134 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:49.634105    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6602187d-5e7c-492c-be4f-e2e0a8d5b794-webhook-cert\") pod \"maas-controller-58fdd44778-vw5qm\" (UID: \"6602187d-5e7c-492c-be4f-e2e0a8d5b794\") " pod="opendatahub/maas-controller-58fdd44778-vw5qm"
Nov 03 01:56:49.634276 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:49.634144    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6602187d-5e7c-492c-be4f-e2e0a8d5b794-metrics-certs\") pod \"maas-controller-58fdd44778-vw5qm\" (UID: \"6602187d-5e7c-492c-be4f-e2e0a8d5b794\") " pod="opendatahub/maas-controller-58fdd44778-vw5qm"
Nov 03 01:56:49.634276 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:49.634248    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcwfw\" (UniqueName: \"kubernetes.io/projected/6602187d-5e7c-492c-be4f-e2e0a8d5b794-kube-api-access-pcwfw\") pod \"maas-controller-58fdd44778-vw5qm\" (UID: \"6602187d-5e7c-492c-be4f-e2e0a8d5b794\") " pod="opendatahub/maas-controller-58fdd44778-vw5qm"
Nov 03 01:56:49.735683 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:49.735642    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6602187d-5e7c-492c-be4f-e2e0a8d5b794-webhook-cert\") pod \"maas-controller-58fdd44778-vw5qm\" (UID: \"6602187d-5e7c-492c-be4f-e2e0a8d5b794\") " pod="opendatahub/maas-controller-58fdd44778-vw5qm"
Nov 03 01:56:49.735858 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:49.735689    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6602187d-5e7c-492c-be4f-e2e0a8d5b794-metrics-certs\") pod \"maas-controller-58fdd44778-vw5qm\" (UID: \"6602187d-5e7c-492c-be4f-e2e0a8d5b794\") " pod="opendatahub/maas-controller-58fdd44778-vw5qm"
Nov 03 01:56:49.735858 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:49.735755    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcwfw\" (UniqueName: \"kubernetes.io/projected/6602187d-5e7c-492c-be4f-e2e0a8d5b794-kube-api-access-pcwfw\") pod \"maas-controller-58fdd44778-vw5qm\" (UID: \"6602187d-5e7c-492c-be4f-e2e0a8d5b794\") " pod="opendatahub/maas-controller-58fdd44778-vw5qm"
Nov 03 01:56:49.735858 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:56:49.735809    5005 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Nov 03 01:56:49.735982 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:56:49.735886    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6602187d-5e7c-492c-be4f-e2e0a8d5b794-webhook-cert podName:6602187d-5e7c-492c-be4f-e2e0a8d5b794 nodeName:}" failed. No retries permitted until 2026-11-03 01:56:50.235865427 +0000 UTC m=+337.250106796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/6602187d-5e7c-492c-be4f-e2e0a8d5b794-webhook-cert") pod "maas-controller-58fdd44778-vw5qm" (UID: "6602187d-5e7c-492c-be4f-e2e0a8d5b794") : secret "maas-controller-webhook-cert" not found
Nov 03 01:56:49.735982 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:56:49.735896    5005 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Nov 03 01:56:49.735982 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:56:49.735976    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6602187d-5e7c-492c-be4f-e2e0a8d5b794-metrics-certs podName:6602187d-5e7c-492c-be4f-e2e0a8d5b794 nodeName:}" failed. No retries permitted until 2026-11-03 01:56:50.235956897 +0000 UTC m=+337.250198248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6602187d-5e7c-492c-be4f-e2e0a8d5b794-metrics-certs") pod "maas-controller-58fdd44778-vw5qm" (UID: "6602187d-5e7c-492c-be4f-e2e0a8d5b794") : secret "maas-controller-metrics-tls" not found
Nov 03 01:56:49.797735 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:49.797644    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcwfw\" (UniqueName: \"kubernetes.io/projected/6602187d-5e7c-492c-be4f-e2e0a8d5b794-kube-api-access-pcwfw\") pod \"maas-controller-58fdd44778-vw5qm\" (UID: \"6602187d-5e7c-492c-be4f-e2e0a8d5b794\") " pod="opendatahub/maas-controller-58fdd44778-vw5qm"
Nov 03 01:56:50.111996 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:50.111921    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-594d9845fc-rcqmw"]
Nov 03 01:56:50.115556 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:50.115539    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-594d9845fc-rcqmw"
Nov 03 01:56:50.185345 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:50.185321    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-594d9845fc-rcqmw"]
Nov 03 01:56:50.239091 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:50.239064    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/368e9ea7-6100-4196-99af-a84e5901adb9-metrics-certs\") pod \"maas-controller-594d9845fc-rcqmw\" (UID: \"368e9ea7-6100-4196-99af-a84e5901adb9\") " pod="opendatahub/maas-controller-594d9845fc-rcqmw"
Nov 03 01:56:50.239205 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:50.239107    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25rqw\" (UniqueName: \"kubernetes.io/projected/368e9ea7-6100-4196-99af-a84e5901adb9-kube-api-access-25rqw\") pod \"maas-controller-594d9845fc-rcqmw\" (UID: \"368e9ea7-6100-4196-99af-a84e5901adb9\") " pod="opendatahub/maas-controller-594d9845fc-rcqmw"
Nov 03 01:56:50.239292 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:50.239273    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/368e9ea7-6100-4196-99af-a84e5901adb9-webhook-cert\") pod \"maas-controller-594d9845fc-rcqmw\" (UID: \"368e9ea7-6100-4196-99af-a84e5901adb9\") " pod="opendatahub/maas-controller-594d9845fc-rcqmw"
Nov 03 01:56:50.239372 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:50.239347    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6602187d-5e7c-492c-be4f-e2e0a8d5b794-webhook-cert\") pod \"maas-controller-58fdd44778-vw5qm\" (UID: \"6602187d-5e7c-492c-be4f-e2e0a8d5b794\") " pod="opendatahub/maas-controller-58fdd44778-vw5qm"
Nov 03 01:56:50.239498 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:50.239453    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6602187d-5e7c-492c-be4f-e2e0a8d5b794-metrics-certs\") pod \"maas-controller-58fdd44778-vw5qm\" (UID: \"6602187d-5e7c-492c-be4f-e2e0a8d5b794\") " pod="opendatahub/maas-controller-58fdd44778-vw5qm"
Nov 03 01:56:50.241890 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:50.241871    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6602187d-5e7c-492c-be4f-e2e0a8d5b794-webhook-cert\") pod \"maas-controller-58fdd44778-vw5qm\" (UID: \"6602187d-5e7c-492c-be4f-e2e0a8d5b794\") " pod="opendatahub/maas-controller-58fdd44778-vw5qm"
Nov 03 01:56:50.242031 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:50.242012    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6602187d-5e7c-492c-be4f-e2e0a8d5b794-metrics-certs\") pod \"maas-controller-58fdd44778-vw5qm\" (UID: \"6602187d-5e7c-492c-be4f-e2e0a8d5b794\") " pod="opendatahub/maas-controller-58fdd44778-vw5qm"
Nov 03 01:56:50.340085 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:50.340055    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/368e9ea7-6100-4196-99af-a84e5901adb9-webhook-cert\") pod \"maas-controller-594d9845fc-rcqmw\" (UID: \"368e9ea7-6100-4196-99af-a84e5901adb9\") " pod="opendatahub/maas-controller-594d9845fc-rcqmw"
Nov 03 01:56:50.340225 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:50.340138    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/368e9ea7-6100-4196-99af-a84e5901adb9-metrics-certs\") pod \"maas-controller-594d9845fc-rcqmw\" (UID: \"368e9ea7-6100-4196-99af-a84e5901adb9\") " pod="opendatahub/maas-controller-594d9845fc-rcqmw"
Nov 03 01:56:50.340225 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:50.340172    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25rqw\" (UniqueName: \"kubernetes.io/projected/368e9ea7-6100-4196-99af-a84e5901adb9-kube-api-access-25rqw\") pod \"maas-controller-594d9845fc-rcqmw\" (UID: \"368e9ea7-6100-4196-99af-a84e5901adb9\") " pod="opendatahub/maas-controller-594d9845fc-rcqmw"
Nov 03 01:56:50.342463 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:50.342438    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/368e9ea7-6100-4196-99af-a84e5901adb9-webhook-cert\") pod \"maas-controller-594d9845fc-rcqmw\" (UID: \"368e9ea7-6100-4196-99af-a84e5901adb9\") " pod="opendatahub/maas-controller-594d9845fc-rcqmw"
Nov 03 01:56:50.342560 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:50.342496    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/368e9ea7-6100-4196-99af-a84e5901adb9-metrics-certs\") pod \"maas-controller-594d9845fc-rcqmw\" (UID: \"368e9ea7-6100-4196-99af-a84e5901adb9\") " pod="opendatahub/maas-controller-594d9845fc-rcqmw"
Nov 03 01:56:50.393404 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:50.393351    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25rqw\" (UniqueName: \"kubernetes.io/projected/368e9ea7-6100-4196-99af-a84e5901adb9-kube-api-access-25rqw\") pod \"maas-controller-594d9845fc-rcqmw\" (UID: \"368e9ea7-6100-4196-99af-a84e5901adb9\") " pod="opendatahub/maas-controller-594d9845fc-rcqmw"
Nov 03 01:56:50.428220 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:50.428200    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-594d9845fc-rcqmw"
Nov 03 01:56:50.430342 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:50.430323    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58fdd44778-vw5qm"
Nov 03 01:56:50.906644 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:50.906614    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58fdd44778-vw5qm"]
Nov 03 01:56:50.908195 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:50.908175    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-594d9845fc-rcqmw"]
Nov 03 01:56:50.913586 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:56:50.913559    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6602187d_5e7c_492c_be4f_e2e0a8d5b794.slice/crio-79254330d403e544785164b2da07dadd792fe1ea0b99c02243108b90aae931d3 WatchSource:0}: Error finding container 79254330d403e544785164b2da07dadd792fe1ea0b99c02243108b90aae931d3: Status 404 returned error can't find the container with id 79254330d403e544785164b2da07dadd792fe1ea0b99c02243108b90aae931d3
Nov 03 01:56:50.964601 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:56:50.964576    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod368e9ea7_6100_4196_99af_a84e5901adb9.slice/crio-2f90e28eec4709257a59c41481515b895fca74f5de7f0e4101d75067b20cfa48 WatchSource:0}: Error finding container 2f90e28eec4709257a59c41481515b895fca74f5de7f0e4101d75067b20cfa48: Status 404 returned error can't find the container with id 2f90e28eec4709257a59c41481515b895fca74f5de7f0e4101d75067b20cfa48
Nov 03 01:56:51.819647 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:51.819607    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58fdd44778-vw5qm" event={"ID":"6602187d-5e7c-492c-be4f-e2e0a8d5b794","Type":"ContainerStarted","Data":"79254330d403e544785164b2da07dadd792fe1ea0b99c02243108b90aae931d3"}
Nov 03 01:56:51.820756 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:51.820729    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-594d9845fc-rcqmw" event={"ID":"368e9ea7-6100-4196-99af-a84e5901adb9","Type":"ContainerStarted","Data":"2f90e28eec4709257a59c41481515b895fca74f5de7f0e4101d75067b20cfa48"}
Nov 03 01:56:58.859910 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:58.859872    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58fdd44778-vw5qm" event={"ID":"6602187d-5e7c-492c-be4f-e2e0a8d5b794","Type":"ContainerStarted","Data":"4365879b8bdddf7e08277ed577a9112b518ec85aa12a9884f20879e224b211aa"}
Nov 03 01:56:58.860381 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:58.859942    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58fdd44778-vw5qm"
Nov 03 01:56:58.861573 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:58.861549    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-594d9845fc-rcqmw" event={"ID":"368e9ea7-6100-4196-99af-a84e5901adb9","Type":"ContainerStarted","Data":"3533fe803415e517d136a91cdc597fea0e956cf08c6f3e768dfd9d8c259ac4f1"}
Nov 03 01:56:58.861724 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:58.861689    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-594d9845fc-rcqmw"
Nov 03 01:56:58.969039 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:58.968993    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-58fdd44778-vw5qm" podStartSLOduration=2.3815932220000002 podStartE2EDuration="9.96898087s" totalImagesPullingTime="7.587387648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:56:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:56:58.967386226 +0000 UTC m=+345.981627594" watchObservedRunningTime="2026-11-03 01:56:58.96898087 +0000 UTC m=+345.983222234"
Nov 03 01:56:59.085342 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:56:59.085290    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-594d9845fc-rcqmw" podStartSLOduration=1.5403687000000001 podStartE2EDuration="9.085277958s" totalImagesPullingTime="7.544909258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:56:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:56:59.08244674 +0000 UTC m=+346.096688114" watchObservedRunningTime="2026-11-03 01:56:59.085277958 +0000 UTC m=+346.099519322"
Nov 03 01:57:00.712136 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:57:00.712082    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 01:57:02.942637 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:02.942599    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp"]
Nov 03 01:57:02.946296 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:02.946277    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp"
Nov 03 01:57:02.952681 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:02.952656    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a46aa4-c8f9-4fdd-a80a-19f100584149-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kflsp\" (UID: \"e3a46aa4-c8f9-4fdd-a80a-19f100584149\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp"
Nov 03 01:57:02.952840 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:02.952819    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e3a46aa4-c8f9-4fdd-a80a-19f100584149-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kflsp\" (UID: \"e3a46aa4-c8f9-4fdd-a80a-19f100584149\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp"
Nov 03 01:57:02.952902 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:02.952854    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q69ww\" (UniqueName: \"kubernetes.io/projected/e3a46aa4-c8f9-4fdd-a80a-19f100584149-kube-api-access-q69ww\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kflsp\" (UID: \"e3a46aa4-c8f9-4fdd-a80a-19f100584149\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp"
Nov 03 01:57:02.953195 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:02.952962    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e3a46aa4-c8f9-4fdd-a80a-19f100584149-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kflsp\" (UID: \"e3a46aa4-c8f9-4fdd-a80a-19f100584149\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp"
Nov 03 01:57:03.007226 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.007195    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 03 01:57:03.007441 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.007238    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-krvrf\""
Nov 03 01:57:03.007537 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.007265    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 03 01:57:03.007537 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.007366    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 03 01:57:03.009432 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.009407    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp"]
Nov 03 01:57:03.053592 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.053562    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e3a46aa4-c8f9-4fdd-a80a-19f100584149-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kflsp\" (UID: \"e3a46aa4-c8f9-4fdd-a80a-19f100584149\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp"
Nov 03 01:57:03.053592 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.053594    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q69ww\" (UniqueName: \"kubernetes.io/projected/e3a46aa4-c8f9-4fdd-a80a-19f100584149-kube-api-access-q69ww\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kflsp\" (UID: \"e3a46aa4-c8f9-4fdd-a80a-19f100584149\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp"
Nov 03 01:57:03.053807 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.053643    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e3a46aa4-c8f9-4fdd-a80a-19f100584149-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kflsp\" (UID: \"e3a46aa4-c8f9-4fdd-a80a-19f100584149\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp"
Nov 03 01:57:03.053807 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.053715    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a46aa4-c8f9-4fdd-a80a-19f100584149-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kflsp\" (UID: \"e3a46aa4-c8f9-4fdd-a80a-19f100584149\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp"
Nov 03 01:57:03.053807 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:57:03.053734    5005 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Nov 03 01:57:03.053807 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:57:03.053778    5005 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Nov 03 01:57:03.054009 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:57:03.053791    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3a46aa4-c8f9-4fdd-a80a-19f100584149-tls-cert podName:e3a46aa4-c8f9-4fdd-a80a-19f100584149 nodeName:}" failed. No retries permitted until 2026-11-03 01:57:03.553772583 +0000 UTC m=+350.568013932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/e3a46aa4-c8f9-4fdd-a80a-19f100584149-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-kflsp" (UID: "e3a46aa4-c8f9-4fdd-a80a-19f100584149") : secret "usage-logs-tenancy-proxy-tls" not found
Nov 03 01:57:03.054009 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:57:03.053852    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3a46aa4-c8f9-4fdd-a80a-19f100584149-proxy-source podName:e3a46aa4-c8f9-4fdd-a80a-19f100584149 nodeName:}" failed. No retries permitted until 2026-11-03 01:57:03.553834719 +0000 UTC m=+350.568076064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/e3a46aa4-c8f9-4fdd-a80a-19f100584149-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-kflsp" (UID: "e3a46aa4-c8f9-4fdd-a80a-19f100584149") : configmap "usage-tenancy-proxy-source" not found
Nov 03 01:57:03.054455 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.054437    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a46aa4-c8f9-4fdd-a80a-19f100584149-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kflsp\" (UID: \"e3a46aa4-c8f9-4fdd-a80a-19f100584149\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp"
Nov 03 01:57:03.110865 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.110828    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q69ww\" (UniqueName: \"kubernetes.io/projected/e3a46aa4-c8f9-4fdd-a80a-19f100584149-kube-api-access-q69ww\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kflsp\" (UID: \"e3a46aa4-c8f9-4fdd-a80a-19f100584149\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp"
Nov 03 01:57:03.220373 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.220309    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-n4klg"]
Nov 03 01:57:03.223843 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.223822    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-n4klg"
Nov 03 01:57:03.256002 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.255975    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9e336ab-f712-4f2c-b897-fc298e3bfb99-cert\") pod \"kserve-controller-manager-776cc66d47-n4klg\" (UID: \"e9e336ab-f712-4f2c-b897-fc298e3bfb99\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n4klg"
Nov 03 01:57:03.256140 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.256026    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwjct\" (UniqueName: \"kubernetes.io/projected/e9e336ab-f712-4f2c-b897-fc298e3bfb99-kube-api-access-xwjct\") pod \"kserve-controller-manager-776cc66d47-n4klg\" (UID: \"e9e336ab-f712-4f2c-b897-fc298e3bfb99\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n4klg"
Nov 03 01:57:03.272832 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.272812    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 03 01:57:03.272931 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.272867    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-w5447\""
Nov 03 01:57:03.281749 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.281727    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-n4klg"]
Nov 03 01:57:03.357421 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.357386    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwjct\" (UniqueName: \"kubernetes.io/projected/e9e336ab-f712-4f2c-b897-fc298e3bfb99-kube-api-access-xwjct\") pod \"kserve-controller-manager-776cc66d47-n4klg\" (UID: \"e9e336ab-f712-4f2c-b897-fc298e3bfb99\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n4klg"
Nov 03 01:57:03.357565 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.357542    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9e336ab-f712-4f2c-b897-fc298e3bfb99-cert\") pod \"kserve-controller-manager-776cc66d47-n4klg\" (UID: \"e9e336ab-f712-4f2c-b897-fc298e3bfb99\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n4klg"
Nov 03 01:57:03.360143 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.360120    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9e336ab-f712-4f2c-b897-fc298e3bfb99-cert\") pod \"kserve-controller-manager-776cc66d47-n4klg\" (UID: \"e9e336ab-f712-4f2c-b897-fc298e3bfb99\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n4klg"
Nov 03 01:57:03.413028 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.412996    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwjct\" (UniqueName: \"kubernetes.io/projected/e9e336ab-f712-4f2c-b897-fc298e3bfb99-kube-api-access-xwjct\") pod \"kserve-controller-manager-776cc66d47-n4klg\" (UID: \"e9e336ab-f712-4f2c-b897-fc298e3bfb99\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n4klg"
Nov 03 01:57:03.440182 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.440154    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-kzmvf"]
Nov 03 01:57:03.443775 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.443748    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-kzmvf"
Nov 03 01:57:03.458703 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.458674    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a529b5-98e0-433d-ad13-382685e55c5f-tls-certs\") pod \"model-serving-api-75bd89fdfc-kzmvf\" (UID: \"f5a529b5-98e0-433d-ad13-382685e55c5f\") " pod="opendatahub/model-serving-api-75bd89fdfc-kzmvf"
Nov 03 01:57:03.458795 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.458765    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6r57\" (UniqueName: \"kubernetes.io/projected/f5a529b5-98e0-433d-ad13-382685e55c5f-kube-api-access-b6r57\") pod \"model-serving-api-75bd89fdfc-kzmvf\" (UID: \"f5a529b5-98e0-433d-ad13-382685e55c5f\") " pod="opendatahub/model-serving-api-75bd89fdfc-kzmvf"
Nov 03 01:57:03.494224 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.494177    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-d2kr4\""
Nov 03 01:57:03.494224 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.494214    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 03 01:57:03.505434 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.505414    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-kzmvf"]
Nov 03 01:57:03.536644 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.536621    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-n4klg"
Nov 03 01:57:03.559589 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.559564    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e3a46aa4-c8f9-4fdd-a80a-19f100584149-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kflsp\" (UID: \"e3a46aa4-c8f9-4fdd-a80a-19f100584149\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp"
Nov 03 01:57:03.559767 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.559626    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a529b5-98e0-433d-ad13-382685e55c5f-tls-certs\") pod \"model-serving-api-75bd89fdfc-kzmvf\" (UID: \"f5a529b5-98e0-433d-ad13-382685e55c5f\") " pod="opendatahub/model-serving-api-75bd89fdfc-kzmvf"
Nov 03 01:57:03.559767 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.559673    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6r57\" (UniqueName: \"kubernetes.io/projected/f5a529b5-98e0-433d-ad13-382685e55c5f-kube-api-access-b6r57\") pod \"model-serving-api-75bd89fdfc-kzmvf\" (UID: \"f5a529b5-98e0-433d-ad13-382685e55c5f\") " pod="opendatahub/model-serving-api-75bd89fdfc-kzmvf"
Nov 03 01:57:03.559767 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.559743    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e3a46aa4-c8f9-4fdd-a80a-19f100584149-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kflsp\" (UID: \"e3a46aa4-c8f9-4fdd-a80a-19f100584149\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp"
Nov 03 01:57:03.560555 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.560534    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e3a46aa4-c8f9-4fdd-a80a-19f100584149-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kflsp\" (UID: \"e3a46aa4-c8f9-4fdd-a80a-19f100584149\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp"
Nov 03 01:57:03.562636 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.562615    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e3a46aa4-c8f9-4fdd-a80a-19f100584149-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kflsp\" (UID: \"e3a46aa4-c8f9-4fdd-a80a-19f100584149\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp"
Nov 03 01:57:03.562845 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.562821    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a529b5-98e0-433d-ad13-382685e55c5f-tls-certs\") pod \"model-serving-api-75bd89fdfc-kzmvf\" (UID: \"f5a529b5-98e0-433d-ad13-382685e55c5f\") " pod="opendatahub/model-serving-api-75bd89fdfc-kzmvf"
Nov 03 01:57:03.580333 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.580309    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-kzgz4"]
Nov 03 01:57:03.585978 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.585961    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-kzgz4"
Nov 03 01:57:03.613755 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.613724    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6r57\" (UniqueName: \"kubernetes.io/projected/f5a529b5-98e0-433d-ad13-382685e55c5f-kube-api-access-b6r57\") pod \"model-serving-api-75bd89fdfc-kzmvf\" (UID: \"f5a529b5-98e0-433d-ad13-382685e55c5f\") " pod="opendatahub/model-serving-api-75bd89fdfc-kzmvf"
Nov 03 01:57:03.636764 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.636184    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-5xr2n\""
Nov 03 01:57:03.636764 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.636191    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 03 01:57:03.636764 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.636315    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 03 01:57:03.638560 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.638538    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-kzgz4"]
Nov 03 01:57:03.660475 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.660453    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a82e336a-5128-4b44-add6-ea01d6179258-metrics-certs\") pod \"odh-model-controller-769875db56-kzgz4\" (UID: \"a82e336a-5128-4b44-add6-ea01d6179258\") " pod="opendatahub/odh-model-controller-769875db56-kzgz4"
Nov 03 01:57:03.660581 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.660513    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a82e336a-5128-4b44-add6-ea01d6179258-cert\") pod \"odh-model-controller-769875db56-kzgz4\" (UID: \"a82e336a-5128-4b44-add6-ea01d6179258\") " pod="opendatahub/odh-model-controller-769875db56-kzgz4"
Nov 03 01:57:03.660726 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.660672    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cbsj\" (UniqueName: \"kubernetes.io/projected/a82e336a-5128-4b44-add6-ea01d6179258-kube-api-access-2cbsj\") pod \"odh-model-controller-769875db56-kzgz4\" (UID: \"a82e336a-5128-4b44-add6-ea01d6179258\") " pod="opendatahub/odh-model-controller-769875db56-kzgz4"
Nov 03 01:57:03.756020 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.755993    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-kzmvf"
Nov 03 01:57:03.761456 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.761436    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cbsj\" (UniqueName: \"kubernetes.io/projected/a82e336a-5128-4b44-add6-ea01d6179258-kube-api-access-2cbsj\") pod \"odh-model-controller-769875db56-kzgz4\" (UID: \"a82e336a-5128-4b44-add6-ea01d6179258\") " pod="opendatahub/odh-model-controller-769875db56-kzgz4"
Nov 03 01:57:03.761531 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.761488    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a82e336a-5128-4b44-add6-ea01d6179258-metrics-certs\") pod \"odh-model-controller-769875db56-kzgz4\" (UID: \"a82e336a-5128-4b44-add6-ea01d6179258\") " pod="opendatahub/odh-model-controller-769875db56-kzgz4"
Nov 03 01:57:03.761573 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.761538    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a82e336a-5128-4b44-add6-ea01d6179258-cert\") pod \"odh-model-controller-769875db56-kzgz4\" (UID: \"a82e336a-5128-4b44-add6-ea01d6179258\") " pod="opendatahub/odh-model-controller-769875db56-kzgz4"
Nov 03 01:57:03.764107 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.764078    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a82e336a-5128-4b44-add6-ea01d6179258-cert\") pod \"odh-model-controller-769875db56-kzgz4\" (UID: \"a82e336a-5128-4b44-add6-ea01d6179258\") " pod="opendatahub/odh-model-controller-769875db56-kzgz4"
Nov 03 01:57:03.764164 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.764145    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a82e336a-5128-4b44-add6-ea01d6179258-metrics-certs\") pod \"odh-model-controller-769875db56-kzgz4\" (UID: \"a82e336a-5128-4b44-add6-ea01d6179258\") " pod="opendatahub/odh-model-controller-769875db56-kzgz4"
Nov 03 01:57:03.815083 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.815054    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cbsj\" (UniqueName: \"kubernetes.io/projected/a82e336a-5128-4b44-add6-ea01d6179258-kube-api-access-2cbsj\") pod \"odh-model-controller-769875db56-kzgz4\" (UID: \"a82e336a-5128-4b44-add6-ea01d6179258\") " pod="opendatahub/odh-model-controller-769875db56-kzgz4"
Nov 03 01:57:03.859716 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.859662    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp"
Nov 03 01:57:03.907938 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.907912    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-kzgz4"
Nov 03 01:57:03.911898 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:03.911874    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-n4klg"]
Nov 03 01:57:03.923529 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:57:03.923503    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e336ab_f712_4f2c_b897_fc298e3bfb99.slice/crio-491fd49df2e6bf459772a5b10384e97f1f8d9b865f8eee086fd34686b564de63 WatchSource:0}: Error finding container 491fd49df2e6bf459772a5b10384e97f1f8d9b865f8eee086fd34686b564de63: Status 404 returned error can't find the container with id 491fd49df2e6bf459772a5b10384e97f1f8d9b865f8eee086fd34686b564de63
Nov 03 01:57:04.129580 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:04.129512    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-kzmvf"]
Nov 03 01:57:04.134718 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:57:04.134673    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a529b5_98e0_433d_ad13_382685e55c5f.slice/crio-3ec2a00c5cae3bcfd44f178ab470f2d9d7faa71fdb013f1ec17d646cd5ab777d WatchSource:0}: Error finding container 3ec2a00c5cae3bcfd44f178ab470f2d9d7faa71fdb013f1ec17d646cd5ab777d: Status 404 returned error can't find the container with id 3ec2a00c5cae3bcfd44f178ab470f2d9d7faa71fdb013f1ec17d646cd5ab777d
Nov 03 01:57:04.230063 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:04.230029    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp"]
Nov 03 01:57:04.236853 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:57:04.236828    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a46aa4_c8f9_4fdd_a80a_19f100584149.slice/crio-f29e254ff6da4d0ad362b9a0fcd05f590d7829e995616fa960709a179c0f7559 WatchSource:0}: Error finding container f29e254ff6da4d0ad362b9a0fcd05f590d7829e995616fa960709a179c0f7559: Status 404 returned error can't find the container with id f29e254ff6da4d0ad362b9a0fcd05f590d7829e995616fa960709a179c0f7559
Nov 03 01:57:04.294588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:04.294476    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-kzgz4"]
Nov 03 01:57:04.298662 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:57:04.298630    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82e336a_5128_4b44_add6_ea01d6179258.slice/crio-4753cddc3defab173f9ea8bcbd0989f0ce4cea7495f7901052d73a37e58f7d2b WatchSource:0}: Error finding container 4753cddc3defab173f9ea8bcbd0989f0ce4cea7495f7901052d73a37e58f7d2b: Status 404 returned error can't find the container with id 4753cddc3defab173f9ea8bcbd0989f0ce4cea7495f7901052d73a37e58f7d2b
Nov 03 01:57:04.896762 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:04.896722    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-kzmvf" event={"ID":"f5a529b5-98e0-433d-ad13-382685e55c5f","Type":"ContainerStarted","Data":"3ec2a00c5cae3bcfd44f178ab470f2d9d7faa71fdb013f1ec17d646cd5ab777d"}
Nov 03 01:57:04.897874 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:04.897841    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-kzgz4" event={"ID":"a82e336a-5128-4b44-add6-ea01d6179258","Type":"ContainerStarted","Data":"4753cddc3defab173f9ea8bcbd0989f0ce4cea7495f7901052d73a37e58f7d2b"}
Nov 03 01:57:04.898969 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:04.898937    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp" event={"ID":"e3a46aa4-c8f9-4fdd-a80a-19f100584149","Type":"ContainerStarted","Data":"f29e254ff6da4d0ad362b9a0fcd05f590d7829e995616fa960709a179c0f7559"}
Nov 03 01:57:04.900170 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:04.900148    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-n4klg" event={"ID":"e9e336ab-f712-4f2c-b897-fc298e3bfb99","Type":"ContainerStarted","Data":"491fd49df2e6bf459772a5b10384e97f1f8d9b865f8eee086fd34686b564de63"}
Nov 03 01:57:07.918922 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:07.918846    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-kzmvf" event={"ID":"f5a529b5-98e0-433d-ad13-382685e55c5f","Type":"ContainerStarted","Data":"6cd1248f96f44e38bc7b22018a8bf736fd1917b81a2742b275671c5fadbc04fc"}
Nov 03 01:57:07.919423 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:07.918974    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-kzmvf"
Nov 03 01:57:08.027802 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:08.027733    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-kzmvf" podStartSLOduration=1.889003163 podStartE2EDuration="5.027715528s" totalImagesPullingTime="3.138712365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:57:08.025681702 +0000 UTC m=+355.039923067" watchObservedRunningTime="2026-11-03 01:57:08.027715528 +0000 UTC m=+355.041956887"
Nov 03 01:57:09.875723 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:09.874176    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-594d9845fc-rcqmw"
Nov 03 01:57:09.875723 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:09.875187    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58fdd44778-vw5qm"
Nov 03 01:57:10.036521 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:10.035944    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58fdd44778-vw5qm"]
Nov 03 01:57:10.036521 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:10.036197    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-58fdd44778-vw5qm" podUID="6602187d-5e7c-492c-be4f-e2e0a8d5b794" containerName="manager" containerID="cri-o://4365879b8bdddf7e08277ed577a9112b518ec85aa12a9884f20879e224b211aa" gracePeriod=30
Nov 03 01:57:10.352491 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:10.352463    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58fdd44778-vw5qm"
Nov 03 01:57:10.418837 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:10.418723    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6602187d-5e7c-492c-be4f-e2e0a8d5b794-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6602187d-5e7c-492c-be4f-e2e0a8d5b794-metrics-certs\") pod \"6602187d-5e7c-492c-be4f-e2e0a8d5b794\" (UID: \"6602187d-5e7c-492c-be4f-e2e0a8d5b794\") "
Nov 03 01:57:10.418837 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:10.418824    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6602187d-5e7c-492c-be4f-e2e0a8d5b794-webhook-cert\" (UniqueName: \"kubernetes.io/secret/6602187d-5e7c-492c-be4f-e2e0a8d5b794-webhook-cert\") pod \"6602187d-5e7c-492c-be4f-e2e0a8d5b794\" (UID: \"6602187d-5e7c-492c-be4f-e2e0a8d5b794\") "
Nov 03 01:57:10.419081 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:10.418972    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6602187d-5e7c-492c-be4f-e2e0a8d5b794-kube-api-access-pcwfw\" (UniqueName: \"kubernetes.io/projected/6602187d-5e7c-492c-be4f-e2e0a8d5b794-kube-api-access-pcwfw\") pod \"6602187d-5e7c-492c-be4f-e2e0a8d5b794\" (UID: \"6602187d-5e7c-492c-be4f-e2e0a8d5b794\") "
Nov 03 01:57:10.421619 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:10.421584    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6602187d-5e7c-492c-be4f-e2e0a8d5b794-webhook-cert" pod "6602187d-5e7c-492c-be4f-e2e0a8d5b794" (UID: "6602187d-5e7c-492c-be4f-e2e0a8d5b794"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:57:10.422293 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:10.422253    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6602187d-5e7c-492c-be4f-e2e0a8d5b794-kube-api-access-pcwfw" pod "6602187d-5e7c-492c-be4f-e2e0a8d5b794" (UID: "6602187d-5e7c-492c-be4f-e2e0a8d5b794"). InnerVolumeSpecName "kube-api-access-pcwfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:57:10.424796 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:10.424766    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6602187d-5e7c-492c-be4f-e2e0a8d5b794-metrics-certs" pod "6602187d-5e7c-492c-be4f-e2e0a8d5b794" (UID: "6602187d-5e7c-492c-be4f-e2e0a8d5b794"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:57:10.520468 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:10.520431    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcwfw\" (UniqueName: \"kubernetes.io/projected/6602187d-5e7c-492c-be4f-e2e0a8d5b794-kube-api-access-pcwfw\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:57:10.520468 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:10.520460    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6602187d-5e7c-492c-be4f-e2e0a8d5b794-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:57:10.520468 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:10.520470    5005 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6602187d-5e7c-492c-be4f-e2e0a8d5b794-webhook-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:57:10.947712 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:10.942732    5005 generic.go:356] "Generic (PLEG): container finished" podID="6602187d-5e7c-492c-be4f-e2e0a8d5b794" containerID="4365879b8bdddf7e08277ed577a9112b518ec85aa12a9884f20879e224b211aa" exitCode=0
Nov 03 01:57:10.947712 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:10.942797    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58fdd44778-vw5qm" event={"ID":"6602187d-5e7c-492c-be4f-e2e0a8d5b794","Type":"ContainerDied","Data":"4365879b8bdddf7e08277ed577a9112b518ec85aa12a9884f20879e224b211aa"}
Nov 03 01:57:10.947712 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:10.942877    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58fdd44778-vw5qm" event={"ID":"6602187d-5e7c-492c-be4f-e2e0a8d5b794","Type":"ContainerDied","Data":"79254330d403e544785164b2da07dadd792fe1ea0b99c02243108b90aae931d3"}
Nov 03 01:57:10.947712 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:10.942928    5005 scope.go:122] "RemoveContainer" containerID="4365879b8bdddf7e08277ed577a9112b518ec85aa12a9884f20879e224b211aa"
Nov 03 01:57:10.947712 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:10.943076    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58fdd44778-vw5qm"
Nov 03 01:57:10.963817 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:10.963797    5005 scope.go:122] "RemoveContainer" containerID="4365879b8bdddf7e08277ed577a9112b518ec85aa12a9884f20879e224b211aa"
Nov 03 01:57:10.964255 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:57:10.964187    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4365879b8bdddf7e08277ed577a9112b518ec85aa12a9884f20879e224b211aa\": container with ID starting with 4365879b8bdddf7e08277ed577a9112b518ec85aa12a9884f20879e224b211aa not found: ID does not exist" containerID="4365879b8bdddf7e08277ed577a9112b518ec85aa12a9884f20879e224b211aa"
Nov 03 01:57:10.964255 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:10.964226    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4365879b8bdddf7e08277ed577a9112b518ec85aa12a9884f20879e224b211aa"} err="failed to get container status \"4365879b8bdddf7e08277ed577a9112b518ec85aa12a9884f20879e224b211aa\": rpc error: code = NotFound desc = could not find container \"4365879b8bdddf7e08277ed577a9112b518ec85aa12a9884f20879e224b211aa\": container with ID starting with 4365879b8bdddf7e08277ed577a9112b518ec85aa12a9884f20879e224b211aa not found: ID does not exist"
Nov 03 01:57:11.110875 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:11.110838    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58fdd44778-vw5qm"]
Nov 03 01:57:11.149237 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:11.149211    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58fdd44778-vw5qm"]
Nov 03 01:57:11.951433 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:11.951334    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-n4klg" event={"ID":"e9e336ab-f712-4f2c-b897-fc298e3bfb99","Type":"ContainerStarted","Data":"2a7a652a8b618c1415b84f09bd841284086ff9c47f2455270f7f9ed0283fbec0"}
Nov 03 01:57:11.951943 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:11.951549    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-n4klg"
Nov 03 01:57:11.955417 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:11.955388    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-kzgz4" event={"ID":"a82e336a-5128-4b44-add6-ea01d6179258","Type":"ContainerStarted","Data":"eebadf991bb15c0635a43d8a3fcbb834fbf7dfdb7d36c0bd808faa8a2bdcf0eb"}
Nov 03 01:57:11.955566 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:11.955540    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-kzgz4"
Nov 03 01:57:12.080044 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:12.079993    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-n4klg" podStartSLOduration=1.332350072 podStartE2EDuration="9.07997851s" totalImagesPullingTime="7.747628438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:57:12.07788752 +0000 UTC m=+359.092128886" watchObservedRunningTime="2026-11-03 01:57:12.07997851 +0000 UTC m=+359.094219916"
Nov 03 01:57:12.188670 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:12.188616    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-kzgz4" podStartSLOduration=2.414123949 podStartE2EDuration="9.188601086s" totalImagesPullingTime="6.774477137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:57:12.186885401 +0000 UTC m=+359.201126765" watchObservedRunningTime="2026-11-03 01:57:12.188601086 +0000 UTC m=+359.202842446"
Nov 03 01:57:13.110964 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:13.110932    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6602187d-5e7c-492c-be4f-e2e0a8d5b794" path="/var/lib/kubelet/pods/6602187d-5e7c-492c-be4f-e2e0a8d5b794/volumes"
Nov 03 01:57:18.932068 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:18.932034    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-kzmvf"
Nov 03 01:57:22.963433 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:22.963394    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-kzgz4"
Nov 03 01:57:23.021617 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:23.021585    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp" event={"ID":"e3a46aa4-c8f9-4fdd-a80a-19f100584149","Type":"ContainerStarted","Data":"3e0d6ab261074c9bef726054d72dcdb54b6e4f01b552302e7d8b7a49c911cd71"}
Nov 03 01:57:23.021784 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:23.021721    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp"
Nov 03 01:57:23.186994 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:23.186940    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp" podStartSLOduration=3.06658606 podStartE2EDuration="21.18692656s" totalImagesPullingTime="18.1203405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:57:23.184978674 +0000 UTC m=+370.199220051" watchObservedRunningTime="2026-11-03 01:57:23.18692656 +0000 UTC m=+370.201167937"
Nov 03 01:57:30.097652 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.097620    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"]
Nov 03 01:57:30.098066 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.098008    5005 cpu_manager.go:414] "containerMap: removing container" podUID="6602187d-5e7c-492c-be4f-e2e0a8d5b794" containerName="manager"
Nov 03 01:57:30.098066 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.098019    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6602187d-5e7c-492c-be4f-e2e0a8d5b794" containerName="manager"
Nov 03 01:57:30.098150 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.098101    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="6602187d-5e7c-492c-be4f-e2e0a8d5b794" containerName="manager"
Nov 03 01:57:30.105219 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.105198    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"
Nov 03 01:57:30.156679 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.156655    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"]
Nov 03 01:57:30.198281 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.198256    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99bc392-e0a4-4a9c-9209-723fe38069c4-metrics-certs\") pod \"maas-api-97b45d8c9-gszmn\" (UID: \"b99bc392-e0a4-4a9c-9209-723fe38069c4\") " pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"
Nov 03 01:57:30.198390 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.198292    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99bc392-e0a4-4a9c-9209-723fe38069c4-openshift-service-ca\") pod \"maas-api-97b45d8c9-gszmn\" (UID: \"b99bc392-e0a4-4a9c-9209-723fe38069c4\") " pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"
Nov 03 01:57:30.198390 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.198314    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b99bc392-e0a4-4a9c-9209-723fe38069c4-maas-api-tls\") pod \"maas-api-97b45d8c9-gszmn\" (UID: \"b99bc392-e0a4-4a9c-9209-723fe38069c4\") " pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"
Nov 03 01:57:30.198390 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.198373    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qshd9\" (UniqueName: \"kubernetes.io/projected/b99bc392-e0a4-4a9c-9209-723fe38069c4-kube-api-access-qshd9\") pod \"maas-api-97b45d8c9-gszmn\" (UID: \"b99bc392-e0a4-4a9c-9209-723fe38069c4\") " pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"
Nov 03 01:57:30.201074 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.201059    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qv59l\""
Nov 03 01:57:30.201134 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.201059    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 01:57:30.201134 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.201060    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 01:57:30.299046 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.299027    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99bc392-e0a4-4a9c-9209-723fe38069c4-metrics-certs\") pod \"maas-api-97b45d8c9-gszmn\" (UID: \"b99bc392-e0a4-4a9c-9209-723fe38069c4\") " pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"
Nov 03 01:57:30.299149 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.299064    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99bc392-e0a4-4a9c-9209-723fe38069c4-openshift-service-ca\") pod \"maas-api-97b45d8c9-gszmn\" (UID: \"b99bc392-e0a4-4a9c-9209-723fe38069c4\") " pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"
Nov 03 01:57:30.299149 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.299083    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b99bc392-e0a4-4a9c-9209-723fe38069c4-maas-api-tls\") pod \"maas-api-97b45d8c9-gszmn\" (UID: \"b99bc392-e0a4-4a9c-9209-723fe38069c4\") " pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"
Nov 03 01:57:30.299249 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.299191    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qshd9\" (UniqueName: \"kubernetes.io/projected/b99bc392-e0a4-4a9c-9209-723fe38069c4-kube-api-access-qshd9\") pod \"maas-api-97b45d8c9-gszmn\" (UID: \"b99bc392-e0a4-4a9c-9209-723fe38069c4\") " pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"
Nov 03 01:57:30.299669 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.299650    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99bc392-e0a4-4a9c-9209-723fe38069c4-openshift-service-ca\") pod \"maas-api-97b45d8c9-gszmn\" (UID: \"b99bc392-e0a4-4a9c-9209-723fe38069c4\") " pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"
Nov 03 01:57:30.301361 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.301341    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99bc392-e0a4-4a9c-9209-723fe38069c4-metrics-certs\") pod \"maas-api-97b45d8c9-gszmn\" (UID: \"b99bc392-e0a4-4a9c-9209-723fe38069c4\") " pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"
Nov 03 01:57:30.301447 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.301431    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b99bc392-e0a4-4a9c-9209-723fe38069c4-maas-api-tls\") pod \"maas-api-97b45d8c9-gszmn\" (UID: \"b99bc392-e0a4-4a9c-9209-723fe38069c4\") " pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"
Nov 03 01:57:30.351653 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.351602    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qshd9\" (UniqueName: \"kubernetes.io/projected/b99bc392-e0a4-4a9c-9209-723fe38069c4-kube-api-access-qshd9\") pod \"maas-api-97b45d8c9-gszmn\" (UID: \"b99bc392-e0a4-4a9c-9209-723fe38069c4\") " pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"
Nov 03 01:57:30.418164 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.418141    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"
Nov 03 01:57:30.712149 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:57:30.712060    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 01:57:30.785899 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:30.785874    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"]
Nov 03 01:57:30.791773 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:57:30.791740    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99bc392_e0a4_4a9c_9209_723fe38069c4.slice/crio-b15c2317788eed7402547dd303acf2e8cf12cd3e12daf12278c49ccf0c7454b5 WatchSource:0}: Error finding container b15c2317788eed7402547dd303acf2e8cf12cd3e12daf12278c49ccf0c7454b5: Status 404 returned error can't find the container with id b15c2317788eed7402547dd303acf2e8cf12cd3e12daf12278c49ccf0c7454b5
Nov 03 01:57:31.060120 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:31.060076    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn" event={"ID":"b99bc392-e0a4-4a9c-9209-723fe38069c4","Type":"ContainerStarted","Data":"b15c2317788eed7402547dd303acf2e8cf12cd3e12daf12278c49ccf0c7454b5"}
Nov 03 01:57:34.030017 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:34.029984    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kflsp"
Nov 03 01:57:38.097275 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:38.097243    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn" event={"ID":"b99bc392-e0a4-4a9c-9209-723fe38069c4","Type":"ContainerStarted","Data":"9eac9b1aec81ba356d1e004ca03a8a922577ec5877a54a99f7832911a7dca23a"}
Nov 03 01:57:38.097663 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:38.097354    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"
Nov 03 01:57:38.204306 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:38.204259    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn" podStartSLOduration=1.643075115 podStartE2EDuration="8.204244828s" totalImagesPullingTime="6.561169713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:57:38.202725067 +0000 UTC m=+385.216966423" watchObservedRunningTime="2026-11-03 01:57:38.204244828 +0000 UTC m=+385.218486193"
Nov 03 01:57:42.966404 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:42.966373    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-n4klg"
Nov 03 01:57:44.107220 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:44.107194    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"
Nov 03 01:57:45.158832 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:45.158798    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"]
Nov 03 01:57:45.218514 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:45.218484    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"]
Nov 03 01:57:45.218680 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:45.218607    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"
Nov 03 01:57:45.331032 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:45.331001    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jwrd\" (UniqueName: \"kubernetes.io/projected/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-kube-api-access-7jwrd\") pod \"maas-api-f9b4c8d5c-rz97f\" (UID: \"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"
Nov 03 01:57:45.331169 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:45.331038    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-openshift-service-ca\") pod \"maas-api-f9b4c8d5c-rz97f\" (UID: \"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"
Nov 03 01:57:45.331169 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:45.331118    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-metrics-certs\") pod \"maas-api-f9b4c8d5c-rz97f\" (UID: \"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"
Nov 03 01:57:45.331169 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:45.331154    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-maas-api-tls\") pod \"maas-api-f9b4c8d5c-rz97f\" (UID: \"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"
Nov 03 01:57:45.432599 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:45.432535    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jwrd\" (UniqueName: \"kubernetes.io/projected/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-kube-api-access-7jwrd\") pod \"maas-api-f9b4c8d5c-rz97f\" (UID: \"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"
Nov 03 01:57:45.432599 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:45.432580    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-openshift-service-ca\") pod \"maas-api-f9b4c8d5c-rz97f\" (UID: \"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"
Nov 03 01:57:45.432805 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:45.432615    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-metrics-certs\") pod \"maas-api-f9b4c8d5c-rz97f\" (UID: \"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"
Nov 03 01:57:45.432805 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:45.432644    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-maas-api-tls\") pod \"maas-api-f9b4c8d5c-rz97f\" (UID: \"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"
Nov 03 01:57:45.433440 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:45.433414    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-openshift-service-ca\") pod \"maas-api-f9b4c8d5c-rz97f\" (UID: \"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"
Nov 03 01:57:45.435019 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:45.434998    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-metrics-certs\") pod \"maas-api-f9b4c8d5c-rz97f\" (UID: \"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"
Nov 03 01:57:45.435269 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:45.435247    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-maas-api-tls\") pod \"maas-api-f9b4c8d5c-rz97f\" (UID: \"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"
Nov 03 01:57:45.485398 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:45.485370    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jwrd\" (UniqueName: \"kubernetes.io/projected/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-kube-api-access-7jwrd\") pod \"maas-api-f9b4c8d5c-rz97f\" (UID: \"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce\") " pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"
Nov 03 01:57:45.550485 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:45.550462    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"
Nov 03 01:57:45.914883 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:45.914843    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"]
Nov 03 01:57:45.920656 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:57:45.920630    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87dd2b4f_63ef_40be_bf4d_3d8e1f71d0ce.slice/crio-99c227d903131718eea5984b0fe655be80414cdf402d6b676ed83acf58982b42 WatchSource:0}: Error finding container 99c227d903131718eea5984b0fe655be80414cdf402d6b676ed83acf58982b42: Status 404 returned error can't find the container with id 99c227d903131718eea5984b0fe655be80414cdf402d6b676ed83acf58982b42
Nov 03 01:57:46.137028 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:46.136995    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f" event={"ID":"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce","Type":"ContainerStarted","Data":"99c227d903131718eea5984b0fe655be80414cdf402d6b676ed83acf58982b42"}
Nov 03 01:57:47.143690 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:47.143600    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f" event={"ID":"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce","Type":"ContainerStarted","Data":"b4b89f7c010ec494b3d6c7c9b9e3a43fc5abd3829a381d83d6f0a89e6b640776"}
Nov 03 01:57:47.144097 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:47.143801    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"
Nov 03 01:57:47.279867 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:47.279817    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f" podStartSLOduration=1.446537287 podStartE2EDuration="2.279804048s" totalImagesPullingTime="833.266761ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:57:47.277971821 +0000 UTC m=+394.292213185" watchObservedRunningTime="2026-11-03 01:57:47.279804048 +0000 UTC m=+394.294045408"
Nov 03 01:57:53.154259 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:53.154189    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"
Nov 03 01:57:53.287969 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:53.287931    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"]
Nov 03 01:57:53.288185 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:53.288160    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn" podUID="b99bc392-e0a4-4a9c-9209-723fe38069c4" containerName="maas-api" containerID="cri-o://9eac9b1aec81ba356d1e004ca03a8a922577ec5877a54a99f7832911a7dca23a" gracePeriod=30
Nov 03 01:57:54.049157 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:54.049134    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"
Nov 03 01:57:54.182110 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:54.182022    5005 generic.go:356] "Generic (PLEG): container finished" podID="b99bc392-e0a4-4a9c-9209-723fe38069c4" containerID="9eac9b1aec81ba356d1e004ca03a8a922577ec5877a54a99f7832911a7dca23a" exitCode=0
Nov 03 01:57:54.182110 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:54.182074    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"
Nov 03 01:57:54.182624 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:54.182101    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn" event={"ID":"b99bc392-e0a4-4a9c-9209-723fe38069c4","Type":"ContainerDied","Data":"9eac9b1aec81ba356d1e004ca03a8a922577ec5877a54a99f7832911a7dca23a"}
Nov 03 01:57:54.182624 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:54.182158    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn" event={"ID":"b99bc392-e0a4-4a9c-9209-723fe38069c4","Type":"ContainerDied","Data":"b15c2317788eed7402547dd303acf2e8cf12cd3e12daf12278c49ccf0c7454b5"}
Nov 03 01:57:54.182624 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:54.182179    5005 scope.go:122] "RemoveContainer" containerID="9eac9b1aec81ba356d1e004ca03a8a922577ec5877a54a99f7832911a7dca23a"
Nov 03 01:57:54.190753 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:54.190732    5005 scope.go:122] "RemoveContainer" containerID="9eac9b1aec81ba356d1e004ca03a8a922577ec5877a54a99f7832911a7dca23a"
Nov 03 01:57:54.191043 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:57:54.191015    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eac9b1aec81ba356d1e004ca03a8a922577ec5877a54a99f7832911a7dca23a\": container with ID starting with 9eac9b1aec81ba356d1e004ca03a8a922577ec5877a54a99f7832911a7dca23a not found: ID does not exist" containerID="9eac9b1aec81ba356d1e004ca03a8a922577ec5877a54a99f7832911a7dca23a"
Nov 03 01:57:54.191132 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:54.191044    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eac9b1aec81ba356d1e004ca03a8a922577ec5877a54a99f7832911a7dca23a"} err="failed to get container status \"9eac9b1aec81ba356d1e004ca03a8a922577ec5877a54a99f7832911a7dca23a\": rpc error: code = NotFound desc = could not find container \"9eac9b1aec81ba356d1e004ca03a8a922577ec5877a54a99f7832911a7dca23a\": container with ID starting with 9eac9b1aec81ba356d1e004ca03a8a922577ec5877a54a99f7832911a7dca23a not found: ID does not exist"
Nov 03 01:57:54.219359 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:54.219335    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b99bc392-e0a4-4a9c-9209-723fe38069c4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99bc392-e0a4-4a9c-9209-723fe38069c4-metrics-certs\") pod \"b99bc392-e0a4-4a9c-9209-723fe38069c4\" (UID: \"b99bc392-e0a4-4a9c-9209-723fe38069c4\") "
Nov 03 01:57:54.219461 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:54.219381    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b99bc392-e0a4-4a9c-9209-723fe38069c4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b99bc392-e0a4-4a9c-9209-723fe38069c4-maas-api-tls\") pod \"b99bc392-e0a4-4a9c-9209-723fe38069c4\" (UID: \"b99bc392-e0a4-4a9c-9209-723fe38069c4\") "
Nov 03 01:57:54.219461 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:54.219405    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b99bc392-e0a4-4a9c-9209-723fe38069c4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99bc392-e0a4-4a9c-9209-723fe38069c4-openshift-service-ca\") pod \"b99bc392-e0a4-4a9c-9209-723fe38069c4\" (UID: \"b99bc392-e0a4-4a9c-9209-723fe38069c4\") "
Nov 03 01:57:54.219533 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:54.219507    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99bc392-e0a4-4a9c-9209-723fe38069c4-kube-api-access-qshd9\" (UniqueName: \"kubernetes.io/projected/b99bc392-e0a4-4a9c-9209-723fe38069c4-kube-api-access-qshd9\") pod \"b99bc392-e0a4-4a9c-9209-723fe38069c4\" (UID: \"b99bc392-e0a4-4a9c-9209-723fe38069c4\") "
Nov 03 01:57:54.219846 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:54.219824    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b99bc392-e0a4-4a9c-9209-723fe38069c4-openshift-service-ca" pod "b99bc392-e0a4-4a9c-9209-723fe38069c4" (UID: "b99bc392-e0a4-4a9c-9209-723fe38069c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:57:54.221613 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:54.221593    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99bc392-e0a4-4a9c-9209-723fe38069c4-kube-api-access-qshd9" pod "b99bc392-e0a4-4a9c-9209-723fe38069c4" (UID: "b99bc392-e0a4-4a9c-9209-723fe38069c4"). InnerVolumeSpecName "kube-api-access-qshd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:57:54.221712 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:54.221632    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b99bc392-e0a4-4a9c-9209-723fe38069c4-maas-api-tls" pod "b99bc392-e0a4-4a9c-9209-723fe38069c4" (UID: "b99bc392-e0a4-4a9c-9209-723fe38069c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:57:54.221751 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:54.221677    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b99bc392-e0a4-4a9c-9209-723fe38069c4-metrics-certs" pod "b99bc392-e0a4-4a9c-9209-723fe38069c4" (UID: "b99bc392-e0a4-4a9c-9209-723fe38069c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:57:54.320433 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:54.320406    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b99bc392-e0a4-4a9c-9209-723fe38069c4-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:57:54.320433 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:54.320432    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b99bc392-e0a4-4a9c-9209-723fe38069c4-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:57:54.320588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:54.320445    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99bc392-e0a4-4a9c-9209-723fe38069c4-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:57:54.320588 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:54.320458    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qshd9\" (UniqueName: \"kubernetes.io/projected/b99bc392-e0a4-4a9c-9209-723fe38069c4-kube-api-access-qshd9\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:57:54.594797 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:54.594769    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"]
Nov 03 01:57:54.596740 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:54.596720    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-97b45d8c9-gszmn"]
Nov 03 01:57:55.107754 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:57:55.107718    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b99bc392-e0a4-4a9c-9209-723fe38069c4" path="/var/lib/kubelet/pods/b99bc392-e0a4-4a9c-9209-723fe38069c4/volumes"
Nov 03 01:58:00.712008 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:58:00.711971    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 01:58:30.711609 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:58:30.711573    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 01:58:53.987008 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:58:53.986979    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-dn4d4"]
Nov 03 01:58:53.987469 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:58:53.987391    5005 cpu_manager.go:414] "containerMap: removing container" podUID="b99bc392-e0a4-4a9c-9209-723fe38069c4" containerName="maas-api"
Nov 03 01:58:53.987469 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:58:53.987403    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b99bc392-e0a4-4a9c-9209-723fe38069c4" containerName="maas-api"
Nov 03 01:58:53.987548 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:58:53.987470    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="b99bc392-e0a4-4a9c-9209-723fe38069c4" containerName="maas-api"
Nov 03 01:58:53.990793 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:58:53.990770    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-dn4d4"
Nov 03 01:58:54.042305 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:58:54.042283    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 01:58:54.079882 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:58:54.079859    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-268vm\" (UniqueName: \"kubernetes.io/projected/30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2-kube-api-access-268vm\") pod \"payload-processing-55fbbbf9dc-dn4d4\" (UID: \"30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-dn4d4"
Nov 03 01:58:54.079993 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:58:54.079904    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-dn4d4\" (UID: \"30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-dn4d4"
Nov 03 01:58:54.088579 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:58:54.088555    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lxnf7\""
Nov 03 01:58:54.090859 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:58:54.090831    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-dn4d4"]
Nov 03 01:58:54.180474 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:58:54.180443    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-268vm\" (UniqueName: \"kubernetes.io/projected/30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2-kube-api-access-268vm\") pod \"payload-processing-55fbbbf9dc-dn4d4\" (UID: \"30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-dn4d4"
Nov 03 01:58:54.180591 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:58:54.180501    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-dn4d4\" (UID: \"30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-dn4d4"
Nov 03 01:58:54.182531 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:58:54.182512    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-dn4d4\" (UID: \"30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-dn4d4"
Nov 03 01:58:54.233191 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:58:54.233166    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-268vm\" (UniqueName: \"kubernetes.io/projected/30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2-kube-api-access-268vm\") pod \"payload-processing-55fbbbf9dc-dn4d4\" (UID: \"30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-dn4d4"
Nov 03 01:58:54.303143 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:58:54.303122    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-dn4d4"
Nov 03 01:58:54.666487 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:58:54.666421    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-dn4d4"]
Nov 03 01:58:54.672541 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:58:54.672519    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a8658b_0d9d_46cc_a3f5_16fc6fdfd5e2.slice/crio-a2b74b61d11fc22d158e0340cfaca7313a4b756e388cfb08db56372c6058d1dc WatchSource:0}: Error finding container a2b74b61d11fc22d158e0340cfaca7313a4b756e388cfb08db56372c6058d1dc: Status 404 returned error can't find the container with id a2b74b61d11fc22d158e0340cfaca7313a4b756e388cfb08db56372c6058d1dc
Nov 03 01:58:55.483542 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:58:55.483506    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-dn4d4" event={"ID":"30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2","Type":"ContainerStarted","Data":"a2b74b61d11fc22d158e0340cfaca7313a4b756e388cfb08db56372c6058d1dc"}
Nov 03 01:59:00.712345 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:59:00.712308    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 01:59:03.526235 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:03.526197    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-dn4d4" event={"ID":"30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2","Type":"ContainerStarted","Data":"77ddd29f0624b7eca34373df58dfdf8a51b57bd720e8ba378ed57790e046aee8"}
Nov 03 01:59:03.526621 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:03.526346    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-dn4d4"
Nov 03 01:59:03.633414 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:03.633365    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-dn4d4" podStartSLOduration=2.494704891 podStartE2EDuration="10.633352668s" totalImagesPullingTime="8.138647777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:59:03.631317018 +0000 UTC m=+470.645558383" watchObservedRunningTime="2026-11-03 01:59:03.633352668 +0000 UTC m=+470.647594082"
Nov 03 01:59:14.533982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:14.533954    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-dn4d4"
Nov 03 01:59:29.923552 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:29.923456    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"]
Nov 03 01:59:29.927547 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:29.927524    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:29.951628 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:29.951594    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffefebf4-2c90-428e-a018-543169ba806c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm\" (UID: \"ffefebf4-2c90-428e-a018-543169ba806c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:29.951763 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:29.951656    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffefebf4-2c90-428e-a018-543169ba806c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm\" (UID: \"ffefebf4-2c90-428e-a018-543169ba806c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:29.951811 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:29.951788    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffefebf4-2c90-428e-a018-543169ba806c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm\" (UID: \"ffefebf4-2c90-428e-a018-543169ba806c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:29.951851 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:29.951845    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffefebf4-2c90-428e-a018-543169ba806c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm\" (UID: \"ffefebf4-2c90-428e-a018-543169ba806c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:29.951911 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:29.951887    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffefebf4-2c90-428e-a018-543169ba806c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm\" (UID: \"ffefebf4-2c90-428e-a018-543169ba806c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:29.951953 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:29.951939    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zrh8\" (UniqueName: \"kubernetes.io/projected/ffefebf4-2c90-428e-a018-543169ba806c-kube-api-access-2zrh8\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm\" (UID: \"ffefebf4-2c90-428e-a018-543169ba806c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:29.976619 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:29.976594    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 01:59:29.999794 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:29.999772    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 01:59:30.022880 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:30.022858    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nlcr8\""
Nov 03 01:59:30.022987 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:30.022901    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 03 01:59:30.025084 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:30.025065    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"]
Nov 03 01:59:30.052779 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:30.052751    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffefebf4-2c90-428e-a018-543169ba806c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm\" (UID: \"ffefebf4-2c90-428e-a018-543169ba806c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:30.052876 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:30.052806    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffefebf4-2c90-428e-a018-543169ba806c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm\" (UID: \"ffefebf4-2c90-428e-a018-543169ba806c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:30.052876 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:30.052845    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffefebf4-2c90-428e-a018-543169ba806c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm\" (UID: \"ffefebf4-2c90-428e-a018-543169ba806c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:30.052990 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:30.052903    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffefebf4-2c90-428e-a018-543169ba806c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm\" (UID: \"ffefebf4-2c90-428e-a018-543169ba806c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:30.052990 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:30.052926    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffefebf4-2c90-428e-a018-543169ba806c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm\" (UID: \"ffefebf4-2c90-428e-a018-543169ba806c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:30.052990 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:30.052952    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zrh8\" (UniqueName: \"kubernetes.io/projected/ffefebf4-2c90-428e-a018-543169ba806c-kube-api-access-2zrh8\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm\" (UID: \"ffefebf4-2c90-428e-a018-543169ba806c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:30.053134 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:30.053120    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffefebf4-2c90-428e-a018-543169ba806c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm\" (UID: \"ffefebf4-2c90-428e-a018-543169ba806c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:30.053270 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:30.053247    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffefebf4-2c90-428e-a018-543169ba806c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm\" (UID: \"ffefebf4-2c90-428e-a018-543169ba806c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:30.053347 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:30.053331    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffefebf4-2c90-428e-a018-543169ba806c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm\" (UID: \"ffefebf4-2c90-428e-a018-543169ba806c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:30.054985 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:30.054969    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffefebf4-2c90-428e-a018-543169ba806c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm\" (UID: \"ffefebf4-2c90-428e-a018-543169ba806c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:30.055388 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:30.055370    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffefebf4-2c90-428e-a018-543169ba806c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm\" (UID: \"ffefebf4-2c90-428e-a018-543169ba806c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:30.105658 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:30.105632    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zrh8\" (UniqueName: \"kubernetes.io/projected/ffefebf4-2c90-428e-a018-543169ba806c-kube-api-access-2zrh8\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm\" (UID: \"ffefebf4-2c90-428e-a018-543169ba806c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:30.241724 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:30.241666    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:30.607326 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:30.607294    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"]
Nov 03 01:59:30.614016 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:59:30.613988    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffefebf4_2c90_428e_a018_543169ba806c.slice/crio-54df85a3305418fdd7328cc466a04c535a08d25223d767bed3f516c407711553 WatchSource:0}: Error finding container 54df85a3305418fdd7328cc466a04c535a08d25223d767bed3f516c407711553: Status 404 returned error can't find the container with id 54df85a3305418fdd7328cc466a04c535a08d25223d767bed3f516c407711553
Nov 03 01:59:30.659922 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:30.659894    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm" event={"ID":"ffefebf4-2c90-428e-a018-543169ba806c","Type":"ContainerStarted","Data":"54df85a3305418fdd7328cc466a04c535a08d25223d767bed3f516c407711553"}
Nov 03 01:59:30.710947 ip-10-0-139-26 kubenswrapper[5005]: E1103 01:59:30.710915    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 01:59:33.678765 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:33.678725    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm" event={"ID":"ffefebf4-2c90-428e-a018-543169ba806c","Type":"ContainerStarted","Data":"3ee9d6ad69a8910bea9db8ec494dfea1714b12f8696ffe6998be6202707011d4"}
Nov 03 01:59:33.786721 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:33.786636    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm" podStartSLOduration=2.42098619 podStartE2EDuration="4.786622668s" totalImagesPullingTime="2.365636478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:59:33.786027203 +0000 UTC m=+500.800268571" watchObservedRunningTime="2026-11-03 01:59:33.786622668 +0000 UTC m=+500.800864032"
Nov 03 01:59:34.242203 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:34.242170    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:40.071989 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.071946    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"]
Nov 03 01:59:40.076130 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.076102    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:40.127295 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.127266    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91c8d98e-c78f-4749-8a24-86d28672ddbb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2dzf7\" (UID: \"91c8d98e-c78f-4749-8a24-86d28672ddbb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:40.127447 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.127306    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rkpp\" (UniqueName: \"kubernetes.io/projected/91c8d98e-c78f-4749-8a24-86d28672ddbb-kube-api-access-8rkpp\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2dzf7\" (UID: \"91c8d98e-c78f-4749-8a24-86d28672ddbb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:40.127516 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.127434    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91c8d98e-c78f-4749-8a24-86d28672ddbb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2dzf7\" (UID: \"91c8d98e-c78f-4749-8a24-86d28672ddbb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:40.127516 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.127486    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c8d98e-c78f-4749-8a24-86d28672ddbb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2dzf7\" (UID: \"91c8d98e-c78f-4749-8a24-86d28672ddbb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:40.127602 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.127581    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91c8d98e-c78f-4749-8a24-86d28672ddbb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2dzf7\" (UID: \"91c8d98e-c78f-4749-8a24-86d28672ddbb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:40.127640 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.127624    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91c8d98e-c78f-4749-8a24-86d28672ddbb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2dzf7\" (UID: \"91c8d98e-c78f-4749-8a24-86d28672ddbb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:40.176013 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.175986    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 03 01:59:40.178105 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.178081    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"]
Nov 03 01:59:40.228947 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.228920    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91c8d98e-c78f-4749-8a24-86d28672ddbb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2dzf7\" (UID: \"91c8d98e-c78f-4749-8a24-86d28672ddbb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:40.229047 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.228959    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rkpp\" (UniqueName: \"kubernetes.io/projected/91c8d98e-c78f-4749-8a24-86d28672ddbb-kube-api-access-8rkpp\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2dzf7\" (UID: \"91c8d98e-c78f-4749-8a24-86d28672ddbb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:40.229047 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.229018    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91c8d98e-c78f-4749-8a24-86d28672ddbb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2dzf7\" (UID: \"91c8d98e-c78f-4749-8a24-86d28672ddbb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:40.229158 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.229054    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c8d98e-c78f-4749-8a24-86d28672ddbb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2dzf7\" (UID: \"91c8d98e-c78f-4749-8a24-86d28672ddbb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:40.229158 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.229083    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91c8d98e-c78f-4749-8a24-86d28672ddbb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2dzf7\" (UID: \"91c8d98e-c78f-4749-8a24-86d28672ddbb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:40.229158 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.229125    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91c8d98e-c78f-4749-8a24-86d28672ddbb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2dzf7\" (UID: \"91c8d98e-c78f-4749-8a24-86d28672ddbb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:40.229382 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.229363    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91c8d98e-c78f-4749-8a24-86d28672ddbb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2dzf7\" (UID: \"91c8d98e-c78f-4749-8a24-86d28672ddbb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:40.229432 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.229381    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c8d98e-c78f-4749-8a24-86d28672ddbb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2dzf7\" (UID: \"91c8d98e-c78f-4749-8a24-86d28672ddbb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:40.229432 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.229418    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91c8d98e-c78f-4749-8a24-86d28672ddbb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2dzf7\" (UID: \"91c8d98e-c78f-4749-8a24-86d28672ddbb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:40.231188 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.231164    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91c8d98e-c78f-4749-8a24-86d28672ddbb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2dzf7\" (UID: \"91c8d98e-c78f-4749-8a24-86d28672ddbb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:40.231493 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.231476    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91c8d98e-c78f-4749-8a24-86d28672ddbb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2dzf7\" (UID: \"91c8d98e-c78f-4749-8a24-86d28672ddbb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:40.241982 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.241962    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:40.254076 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.254060    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:40.281513 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.281493    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rkpp\" (UniqueName: \"kubernetes.io/projected/91c8d98e-c78f-4749-8a24-86d28672ddbb-kube-api-access-8rkpp\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2dzf7\" (UID: \"91c8d98e-c78f-4749-8a24-86d28672ddbb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:40.389769 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.389690    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:40.734592 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.734527    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm"
Nov 03 01:59:40.757652 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:40.757624    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"]
Nov 03 01:59:40.764288 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:59:40.764265    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c8d98e_c78f_4749_8a24_86d28672ddbb.slice/crio-3a33755f95dcab355d034a5c9a0682447bf64f9f68af8cec72560565231df3e0 WatchSource:0}: Error finding container 3a33755f95dcab355d034a5c9a0682447bf64f9f68af8cec72560565231df3e0: Status 404 returned error can't find the container with id 3a33755f95dcab355d034a5c9a0682447bf64f9f68af8cec72560565231df3e0
Nov 03 01:59:41.731990 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:41.731907    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7" event={"ID":"91c8d98e-c78f-4749-8a24-86d28672ddbb","Type":"ContainerStarted","Data":"feb2c2b2625f99ca3650937606a7817b82949ea7ff7ef5d7fde99e76d4960942"}
Nov 03 01:59:41.731990 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:41.731967    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7" event={"ID":"91c8d98e-c78f-4749-8a24-86d28672ddbb","Type":"ContainerStarted","Data":"3a33755f95dcab355d034a5c9a0682447bf64f9f68af8cec72560565231df3e0"}
Nov 03 01:59:41.839277 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:41.839230    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7" podStartSLOduration=1.283526468 podStartE2EDuration="1.83921665s" totalImagesPullingTime="555.690182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:59:41.83788033 +0000 UTC m=+508.852121719" watchObservedRunningTime="2026-11-03 01:59:41.83921665 +0000 UTC m=+508.853458047"
Nov 03 01:59:42.390776 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:42.390749    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:50.390783 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:50.390736    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:50.402985 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:50.402954    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:50.789893 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:50.789868    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2dzf7"
Nov 03 01:59:52.992281 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:52.992245    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-44tnl"]
Nov 03 01:59:52.998876 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:52.998846    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 01:59:53.029044 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.029022    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cff8b24c-e3ef-4500-859c-528ffdf39d1b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-44tnl\" (UID: \"cff8b24c-e3ef-4500-859c-528ffdf39d1b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 01:59:53.029144 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.029073    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cff8b24c-e3ef-4500-859c-528ffdf39d1b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-44tnl\" (UID: \"cff8b24c-e3ef-4500-859c-528ffdf39d1b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 01:59:53.029144 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.029094    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cff8b24c-e3ef-4500-859c-528ffdf39d1b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-44tnl\" (UID: \"cff8b24c-e3ef-4500-859c-528ffdf39d1b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 01:59:53.029220 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.029191    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br8cr\" (UniqueName: \"kubernetes.io/projected/cff8b24c-e3ef-4500-859c-528ffdf39d1b-kube-api-access-br8cr\") pod \"echo-simulator-kserve-64b6f8f96c-44tnl\" (UID: \"cff8b24c-e3ef-4500-859c-528ffdf39d1b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 01:59:53.029271 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.029256    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cff8b24c-e3ef-4500-859c-528ffdf39d1b-home\") pod \"echo-simulator-kserve-64b6f8f96c-44tnl\" (UID: \"cff8b24c-e3ef-4500-859c-528ffdf39d1b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 01:59:53.029302 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.029285    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cff8b24c-e3ef-4500-859c-528ffdf39d1b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-44tnl\" (UID: \"cff8b24c-e3ef-4500-859c-528ffdf39d1b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 01:59:53.093765 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.093729    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 03 01:59:53.095849 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.095827    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-44tnl"]
Nov 03 01:59:53.129966 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.129945    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cff8b24c-e3ef-4500-859c-528ffdf39d1b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-44tnl\" (UID: \"cff8b24c-e3ef-4500-859c-528ffdf39d1b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 01:59:53.130061 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.130002    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cff8b24c-e3ef-4500-859c-528ffdf39d1b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-44tnl\" (UID: \"cff8b24c-e3ef-4500-859c-528ffdf39d1b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 01:59:53.130061 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.130025    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cff8b24c-e3ef-4500-859c-528ffdf39d1b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-44tnl\" (UID: \"cff8b24c-e3ef-4500-859c-528ffdf39d1b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 01:59:53.130061 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.130042    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br8cr\" (UniqueName: \"kubernetes.io/projected/cff8b24c-e3ef-4500-859c-528ffdf39d1b-kube-api-access-br8cr\") pod \"echo-simulator-kserve-64b6f8f96c-44tnl\" (UID: \"cff8b24c-e3ef-4500-859c-528ffdf39d1b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 01:59:53.130061 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.130059    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cff8b24c-e3ef-4500-859c-528ffdf39d1b-home\") pod \"echo-simulator-kserve-64b6f8f96c-44tnl\" (UID: \"cff8b24c-e3ef-4500-859c-528ffdf39d1b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 01:59:53.130245 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.130078    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cff8b24c-e3ef-4500-859c-528ffdf39d1b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-44tnl\" (UID: \"cff8b24c-e3ef-4500-859c-528ffdf39d1b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 01:59:53.130439 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.130417    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cff8b24c-e3ef-4500-859c-528ffdf39d1b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-44tnl\" (UID: \"cff8b24c-e3ef-4500-859c-528ffdf39d1b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 01:59:53.130509 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.130438    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cff8b24c-e3ef-4500-859c-528ffdf39d1b-home\") pod \"echo-simulator-kserve-64b6f8f96c-44tnl\" (UID: \"cff8b24c-e3ef-4500-859c-528ffdf39d1b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 01:59:53.130509 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.130483    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cff8b24c-e3ef-4500-859c-528ffdf39d1b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-44tnl\" (UID: \"cff8b24c-e3ef-4500-859c-528ffdf39d1b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 01:59:53.132080 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.132057    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cff8b24c-e3ef-4500-859c-528ffdf39d1b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-44tnl\" (UID: \"cff8b24c-e3ef-4500-859c-528ffdf39d1b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 01:59:53.132366 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.132348    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cff8b24c-e3ef-4500-859c-528ffdf39d1b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-44tnl\" (UID: \"cff8b24c-e3ef-4500-859c-528ffdf39d1b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 01:59:53.182597 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.182577    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br8cr\" (UniqueName: \"kubernetes.io/projected/cff8b24c-e3ef-4500-859c-528ffdf39d1b-kube-api-access-br8cr\") pod \"echo-simulator-kserve-64b6f8f96c-44tnl\" (UID: \"cff8b24c-e3ef-4500-859c-528ffdf39d1b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 01:59:53.312817 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.312796    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 01:59:53.683959 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.683892    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-44tnl"]
Nov 03 01:59:53.691103 ip-10-0-139-26 kubenswrapper[5005]: W1103 01:59:53.691076    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff8b24c_e3ef_4500_859c_528ffdf39d1b.slice/crio-f256bbd160e8d4c589a3e3e30425cd972136cfb87da2ee9db7ad47717e0c9478 WatchSource:0}: Error finding container f256bbd160e8d4c589a3e3e30425cd972136cfb87da2ee9db7ad47717e0c9478: Status 404 returned error can't find the container with id f256bbd160e8d4c589a3e3e30425cd972136cfb87da2ee9db7ad47717e0c9478
Nov 03 01:59:53.796002 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:53.795977    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl" event={"ID":"cff8b24c-e3ef-4500-859c-528ffdf39d1b","Type":"ContainerStarted","Data":"f256bbd160e8d4c589a3e3e30425cd972136cfb87da2ee9db7ad47717e0c9478"}
Nov 03 01:59:56.813018 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:56.812976    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl" event={"ID":"cff8b24c-e3ef-4500-859c-528ffdf39d1b","Type":"ContainerStarted","Data":"b9de4ad1f0195814ec2094de04830cb1fdf608beba7eb6726d3f96de89d09a75"}
Nov 03 01:59:56.921155 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:56.921106    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl" podStartSLOduration=2.382798275 podStartE2EDuration="4.921090361s" totalImagesPullingTime="2.538292086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:59:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:59:56.920524273 +0000 UTC m=+523.934765638" watchObservedRunningTime="2026-11-03 01:59:56.921090361 +0000 UTC m=+523.935331726"
Nov 03 01:59:57.313453 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:57.313422    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 01:59:59.789147 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:59.789107    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"]
Nov 03 01:59:59.794642 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:59.794616    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 01:59:59.846376 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:59.846345    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"]
Nov 03 01:59:59.885389 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:59.885363    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d2qj\" (UniqueName: \"kubernetes.io/projected/96c17dd0-a778-4d4d-851f-551c152453ef-kube-api-access-8d2qj\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr\" (UID: \"96c17dd0-a778-4d4d-851f-551c152453ef\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 01:59:59.885488 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:59.885402    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96c17dd0-a778-4d4d-851f-551c152453ef-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr\" (UID: \"96c17dd0-a778-4d4d-851f-551c152453ef\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 01:59:59.885488 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:59.885425    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96c17dd0-a778-4d4d-851f-551c152453ef-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr\" (UID: \"96c17dd0-a778-4d4d-851f-551c152453ef\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 01:59:59.885586 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:59.885564    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96c17dd0-a778-4d4d-851f-551c152453ef-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr\" (UID: \"96c17dd0-a778-4d4d-851f-551c152453ef\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 01:59:59.885625 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:59.885599    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96c17dd0-a778-4d4d-851f-551c152453ef-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr\" (UID: \"96c17dd0-a778-4d4d-851f-551c152453ef\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 01:59:59.885625 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:59.885615    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96c17dd0-a778-4d4d-851f-551c152453ef-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr\" (UID: \"96c17dd0-a778-4d4d-851f-551c152453ef\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 01:59:59.889741 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:59.889724    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 03 01:59:59.986423 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:59.986400    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96c17dd0-a778-4d4d-851f-551c152453ef-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr\" (UID: \"96c17dd0-a778-4d4d-851f-551c152453ef\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 01:59:59.986514 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:59.986428    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96c17dd0-a778-4d4d-851f-551c152453ef-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr\" (UID: \"96c17dd0-a778-4d4d-851f-551c152453ef\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 01:59:59.986514 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:59.986464    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d2qj\" (UniqueName: \"kubernetes.io/projected/96c17dd0-a778-4d4d-851f-551c152453ef-kube-api-access-8d2qj\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr\" (UID: \"96c17dd0-a778-4d4d-851f-551c152453ef\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 01:59:59.986514 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:59.986496    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96c17dd0-a778-4d4d-851f-551c152453ef-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr\" (UID: \"96c17dd0-a778-4d4d-851f-551c152453ef\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 01:59:59.986638 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:59.986519    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96c17dd0-a778-4d4d-851f-551c152453ef-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr\" (UID: \"96c17dd0-a778-4d4d-851f-551c152453ef\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 01:59:59.986638 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:59.986579    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96c17dd0-a778-4d4d-851f-551c152453ef-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr\" (UID: \"96c17dd0-a778-4d4d-851f-551c152453ef\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 01:59:59.986946 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:59.986913    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96c17dd0-a778-4d4d-851f-551c152453ef-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr\" (UID: \"96c17dd0-a778-4d4d-851f-551c152453ef\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 01:59:59.986946 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:59.986937    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96c17dd0-a778-4d4d-851f-551c152453ef-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr\" (UID: \"96c17dd0-a778-4d4d-851f-551c152453ef\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 01:59:59.987098 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:59.986969    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96c17dd0-a778-4d4d-851f-551c152453ef-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr\" (UID: \"96c17dd0-a778-4d4d-851f-551c152453ef\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 01:59:59.988715 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:59.988681    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96c17dd0-a778-4d4d-851f-551c152453ef-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr\" (UID: \"96c17dd0-a778-4d4d-851f-551c152453ef\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 01:59:59.989067 ip-10-0-139-26 kubenswrapper[5005]: I1103 01:59:59.989047    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96c17dd0-a778-4d4d-851f-551c152453ef-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr\" (UID: \"96c17dd0-a778-4d4d-851f-551c152453ef\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 02:00:00.041143 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:00.041088    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d2qj\" (UniqueName: \"kubernetes.io/projected/96c17dd0-a778-4d4d-851f-551c152453ef-kube-api-access-8d2qj\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr\" (UID: \"96c17dd0-a778-4d4d-851f-551c152453ef\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 02:00:00.108520 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:00.108498    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 02:00:00.477335 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:00.477264    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"]
Nov 03 02:00:00.482896 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:00:00.482853    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96c17dd0_a778_4d4d_851f_551c152453ef.slice/crio-6a934a11d56484f07c07b0e9ed8f8d731fdd6bb8d1222db99e6bd130cb4c4e3a WatchSource:0}: Error finding container 6a934a11d56484f07c07b0e9ed8f8d731fdd6bb8d1222db99e6bd130cb4c4e3a: Status 404 returned error can't find the container with id 6a934a11d56484f07c07b0e9ed8f8d731fdd6bb8d1222db99e6bd130cb4c4e3a
Nov 03 02:00:00.710665 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:00:00.710633    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:00:00.835307 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:00.835277    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr" event={"ID":"96c17dd0-a778-4d4d-851f-551c152453ef","Type":"ContainerStarted","Data":"6a934a11d56484f07c07b0e9ed8f8d731fdd6bb8d1222db99e6bd130cb4c4e3a"}
Nov 03 02:00:01.841451 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:01.841418    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr" event={"ID":"96c17dd0-a778-4d4d-851f-551c152453ef","Type":"ContainerStarted","Data":"fa548d0e9a177aa92715349f218f0ead311a43248d76877759c0e522a11a9fc9"}
Nov 03 02:00:01.949745 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:01.949686    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr" podStartSLOduration=2.044210383 podStartE2EDuration="2.949673753s" totalImagesPullingTime="905.46337ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:59:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:01.947463476 +0000 UTC m=+528.961704840" watchObservedRunningTime="2026-11-03 02:00:01.949673753 +0000 UTC m=+528.963915179"
Nov 03 02:00:02.109164 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:02.109108    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 02:00:03.313191 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.313160    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 02:00:03.329931 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.329907    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 02:00:03.485085 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.484746    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"]
Nov 03 02:00:03.488463 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.488447    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.536915 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.536891    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fwpk8\""
Nov 03 02:00:03.546522 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.546502    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"]
Nov 03 02:00:03.614896 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.614821    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee540784-ffc0-40e2-be04-d8c13a2521b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.614896 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.614861    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee540784-ffc0-40e2-be04-d8c13a2521b1-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.615036 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.614898    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee540784-ffc0-40e2-be04-d8c13a2521b1-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.615036 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.615024    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee540784-ffc0-40e2-be04-d8c13a2521b1-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.615114 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.615050    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee540784-ffc0-40e2-be04-d8c13a2521b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.615114 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.615073    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee540784-ffc0-40e2-be04-d8c13a2521b1-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.615193 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.615175    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee540784-ffc0-40e2-be04-d8c13a2521b1-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.615231 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.615204    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee540784-ffc0-40e2-be04-d8c13a2521b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.615231 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.615219    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7rnv\" (UniqueName: \"kubernetes.io/projected/ee540784-ffc0-40e2-be04-d8c13a2521b1-kube-api-access-z7rnv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.715993 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.715968    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee540784-ffc0-40e2-be04-d8c13a2521b1-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.716088 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.715996    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee540784-ffc0-40e2-be04-d8c13a2521b1-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.716088 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.716021    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee540784-ffc0-40e2-be04-d8c13a2521b1-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.716088 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.716039    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee540784-ffc0-40e2-be04-d8c13a2521b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.716088 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.716058    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee540784-ffc0-40e2-be04-d8c13a2521b1-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.716265 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.716133    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee540784-ffc0-40e2-be04-d8c13a2521b1-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.716265 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.716160    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee540784-ffc0-40e2-be04-d8c13a2521b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.716265 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.716185    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7rnv\" (UniqueName: \"kubernetes.io/projected/ee540784-ffc0-40e2-be04-d8c13a2521b1-kube-api-access-z7rnv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.716265 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.716246    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee540784-ffc0-40e2-be04-d8c13a2521b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.716465 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.716406    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee540784-ffc0-40e2-be04-d8c13a2521b1-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.716567 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.716550    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee540784-ffc0-40e2-be04-d8c13a2521b1-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.716643 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.716569    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee540784-ffc0-40e2-be04-d8c13a2521b1-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.716643 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.716632    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee540784-ffc0-40e2-be04-d8c13a2521b1-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.716828 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.716807    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee540784-ffc0-40e2-be04-d8c13a2521b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.718432 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.718413    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee540784-ffc0-40e2-be04-d8c13a2521b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.718706 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.718679    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee540784-ffc0-40e2-be04-d8c13a2521b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.770478 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.770445    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee540784-ffc0-40e2-be04-d8c13a2521b1-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.771133 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.771108    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7rnv\" (UniqueName: \"kubernetes.io/projected/ee540784-ffc0-40e2-be04-d8c13a2521b1-kube-api-access-z7rnv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-pd8pt\" (UID: \"ee540784-ffc0-40e2-be04-d8c13a2521b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.800809 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.800791    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:03.868626 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:03.868573    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-44tnl"
Nov 03 02:00:04.220728 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:04.220639    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"]
Nov 03 02:00:04.226963 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:00:04.226931    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee540784_ffc0_40e2_be04_d8c13a2521b1.slice/crio-050467bb36ed058f360ac3e6333e653e333c4ca420dbda6e43c6d027e21eab05 WatchSource:0}: Error finding container 050467bb36ed058f360ac3e6333e653e333c4ca420dbda6e43c6d027e21eab05: Status 404 returned error can't find the container with id 050467bb36ed058f360ac3e6333e653e333c4ca420dbda6e43c6d027e21eab05
Nov 03 02:00:04.229260 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:04.229224    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:00:04.229362 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:04.229312    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:00:04.229362 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:04.229356    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:00:04.859210 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:04.859172    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt" event={"ID":"ee540784-ffc0-40e2-be04-d8c13a2521b1","Type":"ContainerStarted","Data":"e61fef3c7c64512cbaa7126c4ed0bd5606c06aa95dd2ff59ea57cf1e2bf795d6"}
Nov 03 02:00:04.859682 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:04.859234    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt" event={"ID":"ee540784-ffc0-40e2-be04-d8c13a2521b1","Type":"ContainerStarted","Data":"050467bb36ed058f360ac3e6333e653e333c4ca420dbda6e43c6d027e21eab05"}
Nov 03 02:00:04.967207 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:04.967144    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt" podStartSLOduration=1.967129671 podStartE2EDuration="1.967129671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:04.967057912 +0000 UTC m=+531.981299278" watchObservedRunningTime="2026-11-03 02:00:04.967129671 +0000 UTC m=+531.981371036"
Nov 03 02:00:05.814768 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:05.810869    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:05.814768 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:05.811023    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-pd8pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 02:00:05.814768 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:05.811079    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt" podUID="ee540784-ffc0-40e2-be04-d8c13a2521b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 02:00:06.816718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:06.805966    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-pd8pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 02:00:06.816718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:06.806022    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt" podUID="ee540784-ffc0-40e2-be04-d8c13a2521b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 02:00:07.822718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:07.821783    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-pd8pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 02:00:07.822718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:07.821840    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt" podUID="ee540784-ffc0-40e2-be04-d8c13a2521b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 02:00:08.808718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:08.806972    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-pd8pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 02:00:08.808718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:08.807041    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt" podUID="ee540784-ffc0-40e2-be04-d8c13a2521b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 02:00:09.803715 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:09.802984    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-pd8pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 02:00:09.803715 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:09.803047    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt" podUID="ee540784-ffc0-40e2-be04-d8c13a2521b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 02:00:10.122905 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:10.122809    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 02:00:10.135440 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:10.135416    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 02:00:10.801712 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:10.801660    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-pd8pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 02:00:10.801872 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:10.801755    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt" podUID="ee540784-ffc0-40e2-be04-d8c13a2521b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 02:00:10.899073 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:10.899045    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 02:00:12.551878 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:12.551837    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:12.552374 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:12.552140    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:12.552982 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:12.552963    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt"
Nov 03 02:00:18.628709 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.628667    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"]
Nov 03 02:00:18.633872 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.633846    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.746396 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.746371    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"]
Nov 03 02:00:18.756633 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.756613    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.756756 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.756642    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.756756 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.756668    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.756839 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.756780    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.756839 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.756813    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.756912 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.756893    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.756960 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.756941    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.757052 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.757033    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9w27\" (UniqueName: \"kubernetes.io/projected/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-kube-api-access-p9w27\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.757116 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.757087    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.857598 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.857576    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.857716 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.857604    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.857716 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.857637    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.857809 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.857739    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.857809 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.857780    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.857912 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.857821    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.857912 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.857855    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.857912 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.857882    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9w27\" (UniqueName: \"kubernetes.io/projected/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-kube-api-access-p9w27\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.858094 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.857926    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.858260 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.858236    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.858378 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.858285    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.858378 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.858303    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.858378 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.858339    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.858540 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.858393    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.860060 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.860042    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.860433 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.860416    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.910088 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.910035    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.910285 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.910266    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9w27\" (UniqueName: \"kubernetes.io/projected/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-kube-api-access-p9w27\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ml5cs\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:18.947751 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:18.947730    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:19.365379 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:19.365348    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"]
Nov 03 02:00:19.370913 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:00:19.370889    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab48a26_11f9_4f6e_b0ab_8cbe7d34d0b8.slice/crio-624f6fb0f13d6885f56b9c08589b734438774e4c3b0aaa6141f48b8eb1a57be2 WatchSource:0}: Error finding container 624f6fb0f13d6885f56b9c08589b734438774e4c3b0aaa6141f48b8eb1a57be2: Status 404 returned error can't find the container with id 624f6fb0f13d6885f56b9c08589b734438774e4c3b0aaa6141f48b8eb1a57be2
Nov 03 02:00:19.372986 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:19.372952    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:00:19.373080 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:19.373025    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:00:19.373080 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:19.373066    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:00:19.937184 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:19.937149    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs" event={"ID":"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8","Type":"ContainerStarted","Data":"b2bf49fc1c6762b9886c9bcb565760a84e9ce2cb8fff6933c7b2ae98a3544692"}
Nov 03 02:00:19.937566 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:19.937225    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs" event={"ID":"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8","Type":"ContainerStarted","Data":"624f6fb0f13d6885f56b9c08589b734438774e4c3b0aaa6141f48b8eb1a57be2"}
Nov 03 02:00:19.947869 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:19.947844    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:20.051549 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:20.051499    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs" podStartSLOduration=2.051485615 podStartE2EDuration="2.051485615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:20.048948178 +0000 UTC m=+547.063189543" watchObservedRunningTime="2026-11-03 02:00:20.051485615 +0000 UTC m=+547.065727038"
Nov 03 02:00:20.949679 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:20.949644    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ml5cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 02:00:20.949679 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:20.949711    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs" podUID="6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 02:00:21.948063 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:21.948030    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ml5cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 02:00:21.948317 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:21.948086    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs" podUID="6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 02:00:22.948521 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:22.948483    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ml5cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 02:00:22.948967 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:22.948544    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs" podUID="6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 02:00:23.948256 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:23.948221    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ml5cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 02:00:23.948430 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:23.948276    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs" podUID="6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 02:00:24.948828 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:24.948793    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ml5cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 02:00:24.949238 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:24.948846    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs" podUID="6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 02:00:25.948922 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:25.948890    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ml5cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 02:00:25.949278 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:25.948941    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs" podUID="6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 02:00:27.407658 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:27.407628    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:27.408143 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:27.408122    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:27.408906 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:00:27.408886    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:00:30.711758 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:00:30.711728    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:01:00.713932 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:01:00.713851    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:01:11.200167 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:11.200125    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz"]
Nov 03 02:01:11.204317 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:11.204295    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz"
Nov 03 02:01:11.258141 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:11.258099    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz"]
Nov 03 02:01:11.301049 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:11.301017    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-aead6b17\""
Nov 03 02:01:11.301049 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:11.301022    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-aead6b17-dockercfg-kztwx\""
Nov 03 02:01:11.353152 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:11.353103    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm"]
Nov 03 02:01:11.356873 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:11.356852    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm"
Nov 03 02:01:11.368744 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:11.368689    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shlzn\" (UniqueName: \"kubernetes.io/projected/33f9915e-45ca-4ab4-8466-0b4967abed94-kube-api-access-shlzn\") pod \"payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz\" (UID: \"33f9915e-45ca-4ab4-8466-0b4967abed94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz"
Nov 03 02:01:11.368891 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:11.368751    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f9915e-45ca-4ab4-8466-0b4967abed94-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz\" (UID: \"33f9915e-45ca-4ab4-8466-0b4967abed94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz"
Nov 03 02:01:11.410546 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:11.410510    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm"]
Nov 03 02:01:11.469584 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:11.469486    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zr5z\" (UniqueName: \"kubernetes.io/projected/20c97e3c-7ac2-49f5-9391-2ac5f64d8766-kube-api-access-8zr5z\") pod \"payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm\" (UID: \"20c97e3c-7ac2-49f5-9391-2ac5f64d8766\") " pod="openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm"
Nov 03 02:01:11.469805 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:11.469634    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shlzn\" (UniqueName: \"kubernetes.io/projected/33f9915e-45ca-4ab4-8466-0b4967abed94-kube-api-access-shlzn\") pod \"payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz\" (UID: \"33f9915e-45ca-4ab4-8466-0b4967abed94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz"
Nov 03 02:01:11.469805 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:11.469668    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f9915e-45ca-4ab4-8466-0b4967abed94-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz\" (UID: \"33f9915e-45ca-4ab4-8466-0b4967abed94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz"
Nov 03 02:01:11.469805 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:11.469724    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c97e3c-7ac2-49f5-9391-2ac5f64d8766-plugins-config-volume\") pod \"payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm\" (UID: \"20c97e3c-7ac2-49f5-9391-2ac5f64d8766\") " pod="openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm"
Nov 03 02:01:11.470400 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:11.470380    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f9915e-45ca-4ab4-8466-0b4967abed94-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz\" (UID: \"33f9915e-45ca-4ab4-8466-0b4967abed94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz"
Nov 03 02:01:11.522942 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:11.522910    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shlzn\" (UniqueName: \"kubernetes.io/projected/33f9915e-45ca-4ab4-8466-0b4967abed94-kube-api-access-shlzn\") pod \"payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz\" (UID: \"33f9915e-45ca-4ab4-8466-0b4967abed94\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz"
Nov 03 02:01:11.571345 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:11.571307    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c97e3c-7ac2-49f5-9391-2ac5f64d8766-plugins-config-volume\") pod \"payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm\" (UID: \"20c97e3c-7ac2-49f5-9391-2ac5f64d8766\") " pod="openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm"
Nov 03 02:01:11.571521 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:11.571382    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zr5z\" (UniqueName: \"kubernetes.io/projected/20c97e3c-7ac2-49f5-9391-2ac5f64d8766-kube-api-access-8zr5z\") pod \"payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm\" (UID: \"20c97e3c-7ac2-49f5-9391-2ac5f64d8766\") " pod="openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm"
Nov 03 02:01:11.572089 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:11.572069    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c97e3c-7ac2-49f5-9391-2ac5f64d8766-plugins-config-volume\") pod \"payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm\" (UID: \"20c97e3c-7ac2-49f5-9391-2ac5f64d8766\") " pod="openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm"
Nov 03 02:01:11.624469 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:11.624436    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zr5z\" (UniqueName: \"kubernetes.io/projected/20c97e3c-7ac2-49f5-9391-2ac5f64d8766-kube-api-access-8zr5z\") pod \"payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm\" (UID: \"20c97e3c-7ac2-49f5-9391-2ac5f64d8766\") " pod="openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm"
Nov 03 02:01:11.672301 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:11.672266    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm"
Nov 03 02:01:11.817512 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:11.817475    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz"
Nov 03 02:01:12.046020 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:12.045985    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm"]
Nov 03 02:01:12.053330 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:01:12.053291    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c97e3c_7ac2_49f5_9391_2ac5f64d8766.slice/crio-e5fc882b4cf090bd8259221ba00ca3f0daae1565fafbd6a1b01129035533328d WatchSource:0}: Error finding container e5fc882b4cf090bd8259221ba00ca3f0daae1565fafbd6a1b01129035533328d: Status 404 returned error can't find the container with id e5fc882b4cf090bd8259221ba00ca3f0daae1565fafbd6a1b01129035533328d
Nov 03 02:01:12.190312 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:12.190030    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz"]
Nov 03 02:01:12.199294 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:01:12.199257    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f9915e_45ca_4ab4_8466_0b4967abed94.slice/crio-be73f59dac12156b76ce0f0ef05244e3f2284e88d1c842bafc5bee15065b52af WatchSource:0}: Error finding container be73f59dac12156b76ce0f0ef05244e3f2284e88d1c842bafc5bee15065b52af: Status 404 returned error can't find the container with id be73f59dac12156b76ce0f0ef05244e3f2284e88d1c842bafc5bee15065b52af
Nov 03 02:01:12.213312 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:12.213277    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz" event={"ID":"33f9915e-45ca-4ab4-8466-0b4967abed94","Type":"ContainerStarted","Data":"be73f59dac12156b76ce0f0ef05244e3f2284e88d1c842bafc5bee15065b52af"}
Nov 03 02:01:12.214941 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:12.214912    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm" event={"ID":"20c97e3c-7ac2-49f5-9391-2ac5f64d8766","Type":"ContainerStarted","Data":"acc8e85218924eb0f97164e786d650caa55dcccb9f4570f85d8871e5d348b5c7"}
Nov 03 02:01:12.215046 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:12.214977    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm" event={"ID":"20c97e3c-7ac2-49f5-9391-2ac5f64d8766","Type":"ContainerStarted","Data":"e5fc882b4cf090bd8259221ba00ca3f0daae1565fafbd6a1b01129035533328d"}
Nov 03 02:01:12.215046 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:12.215023    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm"
Nov 03 02:01:12.324475 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:12.324358    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm" podStartSLOduration=1.32434145 podStartE2EDuration="1.32434145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:12.322928214 +0000 UTC m=+599.337169562" watchObservedRunningTime="2026-11-03 02:01:12.32434145 +0000 UTC m=+599.338582816"
Nov 03 02:01:13.158780 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:13.158748    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:01:13.159306 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:13.159285    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:01:13.223000 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:13.222964    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz" event={"ID":"33f9915e-45ca-4ab4-8466-0b4967abed94","Type":"ContainerStarted","Data":"665e8e4855874341e21984e753e05237989471702bd6776e18fbd1c4ebff5c87"}
Nov 03 02:01:13.329999 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:13.329946    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz" podStartSLOduration=2.329933812 podStartE2EDuration="2.329933812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:13.329141977 +0000 UTC m=+600.343383342" watchObservedRunningTime="2026-11-03 02:01:13.329933812 +0000 UTC m=+600.344175179"
Nov 03 02:01:13.871273 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:13.871242    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"]
Nov 03 02:01:13.875180 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:13.875155    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"
Nov 03 02:01:13.927067 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:13.927032    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6ac43c49-serving-cert\""
Nov 03 02:01:13.930512 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:13.930470    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"]
Nov 03 02:01:13.998981 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:13.998948    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-maas-api-tls\") pod \"maas-api-e2e-ait-6ac43c49-55df997955-nmn82\" (UID: \"89afc39a-86cd-46e9-bb50-bd3a3e2d0007\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"
Nov 03 02:01:13.999157 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:13.999014    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-metrics-certs\") pod \"maas-api-e2e-ait-6ac43c49-55df997955-nmn82\" (UID: \"89afc39a-86cd-46e9-bb50-bd3a3e2d0007\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"
Nov 03 02:01:13.999157 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:13.999140    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brqjv\" (UniqueName: \"kubernetes.io/projected/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-kube-api-access-brqjv\") pod \"maas-api-e2e-ait-6ac43c49-55df997955-nmn82\" (UID: \"89afc39a-86cd-46e9-bb50-bd3a3e2d0007\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"
Nov 03 02:01:13.999249 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:13.999178    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-openshift-service-ca\") pod \"maas-api-e2e-ait-6ac43c49-55df997955-nmn82\" (UID: \"89afc39a-86cd-46e9-bb50-bd3a3e2d0007\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"
Nov 03 02:01:14.099953 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:14.099912    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brqjv\" (UniqueName: \"kubernetes.io/projected/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-kube-api-access-brqjv\") pod \"maas-api-e2e-ait-6ac43c49-55df997955-nmn82\" (UID: \"89afc39a-86cd-46e9-bb50-bd3a3e2d0007\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"
Nov 03 02:01:14.099953 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:14.099960    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-openshift-service-ca\") pod \"maas-api-e2e-ait-6ac43c49-55df997955-nmn82\" (UID: \"89afc39a-86cd-46e9-bb50-bd3a3e2d0007\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"
Nov 03 02:01:14.100215 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:14.100023    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-maas-api-tls\") pod \"maas-api-e2e-ait-6ac43c49-55df997955-nmn82\" (UID: \"89afc39a-86cd-46e9-bb50-bd3a3e2d0007\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"
Nov 03 02:01:14.100215 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:14.100109    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-metrics-certs\") pod \"maas-api-e2e-ait-6ac43c49-55df997955-nmn82\" (UID: \"89afc39a-86cd-46e9-bb50-bd3a3e2d0007\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"
Nov 03 02:01:14.100215 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:01:14.100138    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-serving-cert: secret "maas-api-e2e-ait-6ac43c49-serving-cert" not found
Nov 03 02:01:14.100215 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:01:14.100210    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-maas-api-tls podName:89afc39a-86cd-46e9-bb50-bd3a3e2d0007 nodeName:}" failed. No retries permitted until 2026-11-03 02:01:14.600188494 +0000 UTC m=+601.614429842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-maas-api-tls") pod "maas-api-e2e-ait-6ac43c49-55df997955-nmn82" (UID: "89afc39a-86cd-46e9-bb50-bd3a3e2d0007") : secret "maas-api-e2e-ait-6ac43c49-serving-cert" not found
Nov 03 02:01:14.100829 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:14.100803    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-openshift-service-ca\") pod \"maas-api-e2e-ait-6ac43c49-55df997955-nmn82\" (UID: \"89afc39a-86cd-46e9-bb50-bd3a3e2d0007\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"
Nov 03 02:01:14.102521 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:14.102503    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-metrics-certs\") pod \"maas-api-e2e-ait-6ac43c49-55df997955-nmn82\" (UID: \"89afc39a-86cd-46e9-bb50-bd3a3e2d0007\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"
Nov 03 02:01:14.153943 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:14.153865    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brqjv\" (UniqueName: \"kubernetes.io/projected/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-kube-api-access-brqjv\") pod \"maas-api-e2e-ait-6ac43c49-55df997955-nmn82\" (UID: \"89afc39a-86cd-46e9-bb50-bd3a3e2d0007\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"
Nov 03 02:01:14.228441 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:14.228396    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz"
Nov 03 02:01:14.605529 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:14.605492    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-maas-api-tls\") pod \"maas-api-e2e-ait-6ac43c49-55df997955-nmn82\" (UID: \"89afc39a-86cd-46e9-bb50-bd3a3e2d0007\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"
Nov 03 02:01:14.608009 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:14.607985    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-maas-api-tls\") pod \"maas-api-e2e-ait-6ac43c49-55df997955-nmn82\" (UID: \"89afc39a-86cd-46e9-bb50-bd3a3e2d0007\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"
Nov 03 02:01:14.789031 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:14.788990    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"
Nov 03 02:01:15.159482 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:15.159446    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"]
Nov 03 02:01:15.165467 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:01:15.165441    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89afc39a_86cd_46e9_bb50_bd3a3e2d0007.slice/crio-5611b0e6ebfb7430563d2eebeea9336e06127d9a586e1c9f728a9fd41bc26f47 WatchSource:0}: Error finding container 5611b0e6ebfb7430563d2eebeea9336e06127d9a586e1c9f728a9fd41bc26f47: Status 404 returned error can't find the container with id 5611b0e6ebfb7430563d2eebeea9336e06127d9a586e1c9f728a9fd41bc26f47
Nov 03 02:01:15.235282 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:15.235244    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82" event={"ID":"89afc39a-86cd-46e9-bb50-bd3a3e2d0007","Type":"ContainerStarted","Data":"5611b0e6ebfb7430563d2eebeea9336e06127d9a586e1c9f728a9fd41bc26f47"}
Nov 03 02:01:15.963534 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:15.963496    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp"]
Nov 03 02:01:15.968551 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:15.968529    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp"
Nov 03 02:01:16.064516 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:16.064471    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6ac43c49\""
Nov 03 02:01:16.064516 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:16.064472    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6ac43c49-dockercfg-9xlbl\""
Nov 03 02:01:16.068400 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:16.068365    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp"]
Nov 03 02:01:16.117491 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:16.117454    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2"]
Nov 03 02:01:16.121006 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:16.120976    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0275219a-ec56-447b-9b83-19ec8008a85f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp\" (UID: \"0275219a-ec56-447b-9b83-19ec8008a85f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp"
Nov 03 02:01:16.121149 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:16.121132    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bjf6\" (UniqueName: \"kubernetes.io/projected/0275219a-ec56-447b-9b83-19ec8008a85f-kube-api-access-8bjf6\") pod \"payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp\" (UID: \"0275219a-ec56-447b-9b83-19ec8008a85f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp"
Nov 03 02:01:16.121285 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:16.121271    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2"
Nov 03 02:01:16.175014 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:16.174686    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2"]
Nov 03 02:01:16.222464 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:16.222383    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bjf6\" (UniqueName: \"kubernetes.io/projected/0275219a-ec56-447b-9b83-19ec8008a85f-kube-api-access-8bjf6\") pod \"payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp\" (UID: \"0275219a-ec56-447b-9b83-19ec8008a85f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp"
Nov 03 02:01:16.222464 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:16.222423    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwqb7\" (UniqueName: \"kubernetes.io/projected/04b6c270-905c-467d-b07d-6d22ebf79a7b-kube-api-access-qwqb7\") pod \"payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2\" (UID: \"04b6c270-905c-467d-b07d-6d22ebf79a7b\") " pod="openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2"
Nov 03 02:01:16.222655 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:16.222486    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0275219a-ec56-447b-9b83-19ec8008a85f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp\" (UID: \"0275219a-ec56-447b-9b83-19ec8008a85f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp"
Nov 03 02:01:16.222655 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:16.222523    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b6c270-905c-467d-b07d-6d22ebf79a7b-plugins-config-volume\") pod \"payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2\" (UID: \"04b6c270-905c-467d-b07d-6d22ebf79a7b\") " pod="openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2"
Nov 03 02:01:16.223324 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:16.223302    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0275219a-ec56-447b-9b83-19ec8008a85f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp\" (UID: \"0275219a-ec56-447b-9b83-19ec8008a85f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp"
Nov 03 02:01:16.275514 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:16.275481    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bjf6\" (UniqueName: \"kubernetes.io/projected/0275219a-ec56-447b-9b83-19ec8008a85f-kube-api-access-8bjf6\") pod \"payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp\" (UID: \"0275219a-ec56-447b-9b83-19ec8008a85f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp"
Nov 03 02:01:16.282284 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:16.282258    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp"
Nov 03 02:01:16.323798 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:16.323765    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwqb7\" (UniqueName: \"kubernetes.io/projected/04b6c270-905c-467d-b07d-6d22ebf79a7b-kube-api-access-qwqb7\") pod \"payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2\" (UID: \"04b6c270-905c-467d-b07d-6d22ebf79a7b\") " pod="openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2"
Nov 03 02:01:16.323955 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:16.323866    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b6c270-905c-467d-b07d-6d22ebf79a7b-plugins-config-volume\") pod \"payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2\" (UID: \"04b6c270-905c-467d-b07d-6d22ebf79a7b\") " pod="openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2"
Nov 03 02:01:16.324502 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:16.324483    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b6c270-905c-467d-b07d-6d22ebf79a7b-plugins-config-volume\") pod \"payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2\" (UID: \"04b6c270-905c-467d-b07d-6d22ebf79a7b\") " pod="openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2"
Nov 03 02:01:16.377859 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:16.377821    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwqb7\" (UniqueName: \"kubernetes.io/projected/04b6c270-905c-467d-b07d-6d22ebf79a7b-kube-api-access-qwqb7\") pod \"payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2\" (UID: \"04b6c270-905c-467d-b07d-6d22ebf79a7b\") " pod="openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2"
Nov 03 02:01:16.434345 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:16.434309    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2"
Nov 03 02:01:16.650527 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:16.650493    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp"]
Nov 03 02:01:16.656956 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:01:16.656930    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0275219a_ec56_447b_9b83_19ec8008a85f.slice/crio-9623b86ac9b4bcacc72a62e8dbeeba7e476f19d22abdf468f828960a92f3f2fe WatchSource:0}: Error finding container 9623b86ac9b4bcacc72a62e8dbeeba7e476f19d22abdf468f828960a92f3f2fe: Status 404 returned error can't find the container with id 9623b86ac9b4bcacc72a62e8dbeeba7e476f19d22abdf468f828960a92f3f2fe
Nov 03 02:01:16.808103 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:16.808074    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2"]
Nov 03 02:01:16.814588 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:01:16.814553    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b6c270_905c_467d_b07d_6d22ebf79a7b.slice/crio-19b32431bdb73ab258b36a5d5c9d6239782f5b5f4b2bb12c1d6f7ee64707d274 WatchSource:0}: Error finding container 19b32431bdb73ab258b36a5d5c9d6239782f5b5f4b2bb12c1d6f7ee64707d274: Status 404 returned error can't find the container with id 19b32431bdb73ab258b36a5d5c9d6239782f5b5f4b2bb12c1d6f7ee64707d274
Nov 03 02:01:17.253471 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:17.253428    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp" event={"ID":"0275219a-ec56-447b-9b83-19ec8008a85f","Type":"ContainerStarted","Data":"37d3b5c9b4d40ab894aa09d1d2d7ec2f14403a4eef3b9c108c2a5ae131b03a30"}
Nov 03 02:01:17.254008 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:17.253493    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp" event={"ID":"0275219a-ec56-447b-9b83-19ec8008a85f","Type":"ContainerStarted","Data":"9623b86ac9b4bcacc72a62e8dbeeba7e476f19d22abdf468f828960a92f3f2fe"}
Nov 03 02:01:17.254008 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:17.253630    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp"
Nov 03 02:01:17.255171 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:17.255148    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2" event={"ID":"04b6c270-905c-467d-b07d-6d22ebf79a7b","Type":"ContainerStarted","Data":"6fe186a005068d8bf04ee723b20ad884ed2989e984a2a8d7c4923b648c5437e5"}
Nov 03 02:01:17.255301 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:17.255206    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2"
Nov 03 02:01:17.255301 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:17.255231    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2" event={"ID":"04b6c270-905c-467d-b07d-6d22ebf79a7b","Type":"ContainerStarted","Data":"19b32431bdb73ab258b36a5d5c9d6239782f5b5f4b2bb12c1d6f7ee64707d274"}
Nov 03 02:01:17.256725 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:17.256687    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82" event={"ID":"89afc39a-86cd-46e9-bb50-bd3a3e2d0007","Type":"ContainerStarted","Data":"de576f33d899a2c84f2095f867e0e155ae2a9ad7cb4c7370657bf27854c409c1"}
Nov 03 02:01:17.256836 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:17.256769    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"
Nov 03 02:01:17.360011 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:17.359951    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp" podStartSLOduration=2.359935548 podStartE2EDuration="2.359935548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:17.358326285 +0000 UTC m=+604.372567650" watchObservedRunningTime="2026-11-03 02:01:17.359935548 +0000 UTC m=+604.374176913"
Nov 03 02:01:17.468313 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:17.468237    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82" podStartSLOduration=3.170170748 podStartE2EDuration="4.468216931s" totalImagesPullingTime="1.298046183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:17.465075553 +0000 UTC m=+604.479316918" watchObservedRunningTime="2026-11-03 02:01:17.468216931 +0000 UTC m=+604.482458299"
Nov 03 02:01:17.573005 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:17.572900    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2" podStartSLOduration=1.572885647 podStartE2EDuration="1.572885647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:17.570292852 +0000 UTC m=+604.584534220" watchObservedRunningTime="2026-11-03 02:01:17.572885647 +0000 UTC m=+604.587127012"
Nov 03 02:01:18.434319 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:18.434284    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9"]
Nov 03 02:01:18.438242 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:18.438222    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9"
Nov 03 02:01:18.489333 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:18.489291    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-080015ba\""
Nov 03 02:01:18.492054 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:18.492023    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9"]
Nov 03 02:01:18.535940 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:18.535909    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-080015ba-dockercfg-jkdzh\""
Nov 03 02:01:18.552957 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:18.552924    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9n57\" (UniqueName: \"kubernetes.io/projected/97f5dd4f-0053-4481-a27c-987237539a88-kube-api-access-t9n57\") pod \"payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9\" (UID: \"97f5dd4f-0053-4481-a27c-987237539a88\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9"
Nov 03 02:01:18.553137 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:18.553050    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f5dd4f-0053-4481-a27c-987237539a88-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9\" (UID: \"97f5dd4f-0053-4481-a27c-987237539a88\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9"
Nov 03 02:01:18.584214 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:18.584181    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79"]
Nov 03 02:01:18.587966 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:18.587942    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79"
Nov 03 02:01:18.641722 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:18.641665    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79"]
Nov 03 02:01:18.653980 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:18.653938    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f5dd4f-0053-4481-a27c-987237539a88-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9\" (UID: \"97f5dd4f-0053-4481-a27c-987237539a88\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9"
Nov 03 02:01:18.654155 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:18.654013    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9n57\" (UniqueName: \"kubernetes.io/projected/97f5dd4f-0053-4481-a27c-987237539a88-kube-api-access-t9n57\") pod \"payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9\" (UID: \"97f5dd4f-0053-4481-a27c-987237539a88\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9"
Nov 03 02:01:18.654675 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:18.654656    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f5dd4f-0053-4481-a27c-987237539a88-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9\" (UID: \"97f5dd4f-0053-4481-a27c-987237539a88\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9"
Nov 03 02:01:18.707562 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:18.707488    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9n57\" (UniqueName: \"kubernetes.io/projected/97f5dd4f-0053-4481-a27c-987237539a88-kube-api-access-t9n57\") pod \"payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9\" (UID: \"97f5dd4f-0053-4481-a27c-987237539a88\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9"
Nov 03 02:01:18.752332 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:18.752300    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9"
Nov 03 02:01:18.754770 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:18.754746    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn2nt\" (UniqueName: \"kubernetes.io/projected/f10d4192-7fdc-4477-af07-2e37036a6595-kube-api-access-xn2nt\") pod \"payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79\" (UID: \"f10d4192-7fdc-4477-af07-2e37036a6595\") " pod="openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79"
Nov 03 02:01:18.754861 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:18.754820    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10d4192-7fdc-4477-af07-2e37036a6595-plugins-config-volume\") pod \"payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79\" (UID: \"f10d4192-7fdc-4477-af07-2e37036a6595\") " pod="openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79"
Nov 03 02:01:18.855566 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:18.855523    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn2nt\" (UniqueName: \"kubernetes.io/projected/f10d4192-7fdc-4477-af07-2e37036a6595-kube-api-access-xn2nt\") pod \"payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79\" (UID: \"f10d4192-7fdc-4477-af07-2e37036a6595\") " pod="openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79"
Nov 03 02:01:18.855800 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:18.855618    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10d4192-7fdc-4477-af07-2e37036a6595-plugins-config-volume\") pod \"payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79\" (UID: \"f10d4192-7fdc-4477-af07-2e37036a6595\") " pod="openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79"
Nov 03 02:01:18.856628 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:18.856600    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10d4192-7fdc-4477-af07-2e37036a6595-plugins-config-volume\") pod \"payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79\" (UID: \"f10d4192-7fdc-4477-af07-2e37036a6595\") " pod="openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79"
Nov 03 02:01:18.908622 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:18.908591    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn2nt\" (UniqueName: \"kubernetes.io/projected/f10d4192-7fdc-4477-af07-2e37036a6595-kube-api-access-xn2nt\") pod \"payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79\" (UID: \"f10d4192-7fdc-4477-af07-2e37036a6595\") " pod="openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79"
Nov 03 02:01:19.030850 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.030806    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"]
Nov 03 02:01:19.035926 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.035902    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"
Nov 03 02:01:19.084379 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.084334    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-7683f606-serving-cert\""
Nov 03 02:01:19.084566 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.084543    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"]
Nov 03 02:01:19.138623 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.138586    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9"]
Nov 03 02:01:19.146169 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:01:19.146140    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f5dd4f_0053_4481_a27c_987237539a88.slice/crio-bd6f8a4e04bf1195495305619c1b39990eb27d0f55d79173e3453c103a07ef4d WatchSource:0}: Error finding container bd6f8a4e04bf1195495305619c1b39990eb27d0f55d79173e3453c103a07ef4d: Status 404 returned error can't find the container with id bd6f8a4e04bf1195495305619c1b39990eb27d0f55d79173e3453c103a07ef4d
Nov 03 02:01:19.158273 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.158244    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/882bac36-975d-43b0-997d-e32a11b91009-metrics-certs\") pod \"maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll\" (UID: \"882bac36-975d-43b0-997d-e32a11b91009\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"
Nov 03 02:01:19.158396 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.158292    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/882bac36-975d-43b0-997d-e32a11b91009-openshift-service-ca\") pod \"maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll\" (UID: \"882bac36-975d-43b0-997d-e32a11b91009\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"
Nov 03 02:01:19.158437 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.158389    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/882bac36-975d-43b0-997d-e32a11b91009-maas-api-tls\") pod \"maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll\" (UID: \"882bac36-975d-43b0-997d-e32a11b91009\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"
Nov 03 02:01:19.158511 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.158495    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg6ws\" (UniqueName: \"kubernetes.io/projected/882bac36-975d-43b0-997d-e32a11b91009-kube-api-access-xg6ws\") pod \"maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll\" (UID: \"882bac36-975d-43b0-997d-e32a11b91009\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"
Nov 03 02:01:19.202424 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.202397    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79"
Nov 03 02:01:19.269528 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.269475    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/882bac36-975d-43b0-997d-e32a11b91009-openshift-service-ca\") pod \"maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll\" (UID: \"882bac36-975d-43b0-997d-e32a11b91009\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"
Nov 03 02:01:19.269762 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.269550    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/882bac36-975d-43b0-997d-e32a11b91009-maas-api-tls\") pod \"maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll\" (UID: \"882bac36-975d-43b0-997d-e32a11b91009\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"
Nov 03 02:01:19.269762 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.269621    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6ws\" (UniqueName: \"kubernetes.io/projected/882bac36-975d-43b0-997d-e32a11b91009-kube-api-access-xg6ws\") pod \"maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll\" (UID: \"882bac36-975d-43b0-997d-e32a11b91009\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"
Nov 03 02:01:19.269762 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.269735    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/882bac36-975d-43b0-997d-e32a11b91009-metrics-certs\") pod \"maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll\" (UID: \"882bac36-975d-43b0-997d-e32a11b91009\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"
Nov 03 02:01:19.283768 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:01:19.279858    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-serving-cert: secret "maas-api-e2e-migrate-7683f606-serving-cert" not found
Nov 03 02:01:19.283768 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:01:19.279961    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/882bac36-975d-43b0-997d-e32a11b91009-maas-api-tls podName:882bac36-975d-43b0-997d-e32a11b91009 nodeName:}" failed. No retries permitted until 2026-11-03 02:01:19.779938265 +0000 UTC m=+606.794179616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/882bac36-975d-43b0-997d-e32a11b91009-maas-api-tls") pod "maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll" (UID: "882bac36-975d-43b0-997d-e32a11b91009") : secret "maas-api-e2e-migrate-7683f606-serving-cert" not found
Nov 03 02:01:19.283768 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.280708    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/882bac36-975d-43b0-997d-e32a11b91009-openshift-service-ca\") pod \"maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll\" (UID: \"882bac36-975d-43b0-997d-e32a11b91009\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"
Nov 03 02:01:19.287134 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.287089    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/882bac36-975d-43b0-997d-e32a11b91009-metrics-certs\") pod \"maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll\" (UID: \"882bac36-975d-43b0-997d-e32a11b91009\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"
Nov 03 02:01:19.305857 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.302296    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9" event={"ID":"97f5dd4f-0053-4481-a27c-987237539a88","Type":"ContainerStarted","Data":"31175b71ef44e286be73f22e4abecd598e1595f6b5b2031f88fb47f871ee17cc"}
Nov 03 02:01:19.305857 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.302364    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9" event={"ID":"97f5dd4f-0053-4481-a27c-987237539a88","Type":"ContainerStarted","Data":"bd6f8a4e04bf1195495305619c1b39990eb27d0f55d79173e3453c103a07ef4d"}
Nov 03 02:01:19.305857 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.302526    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9"
Nov 03 02:01:19.336044 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.336014    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg6ws\" (UniqueName: \"kubernetes.io/projected/882bac36-975d-43b0-997d-e32a11b91009-kube-api-access-xg6ws\") pod \"maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll\" (UID: \"882bac36-975d-43b0-997d-e32a11b91009\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"
Nov 03 02:01:19.409355 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.409300    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9" podStartSLOduration=1.4092839480000001 podStartE2EDuration="1.409283948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:19.407427877 +0000 UTC m=+606.421669235" watchObservedRunningTime="2026-11-03 02:01:19.409283948 +0000 UTC m=+606.423525314"
Nov 03 02:01:19.595040 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.594947    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79"]
Nov 03 02:01:19.600755 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:01:19.600726    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10d4192_7fdc_4477_af07_2e37036a6595.slice/crio-07edb1b71ce599393af28a94580a32241933865e8ead31c5e50ff67e49c7f4df WatchSource:0}: Error finding container 07edb1b71ce599393af28a94580a32241933865e8ead31c5e50ff67e49c7f4df: Status 404 returned error can't find the container with id 07edb1b71ce599393af28a94580a32241933865e8ead31c5e50ff67e49c7f4df
Nov 03 02:01:19.875215 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.875121    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/882bac36-975d-43b0-997d-e32a11b91009-maas-api-tls\") pod \"maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll\" (UID: \"882bac36-975d-43b0-997d-e32a11b91009\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"
Nov 03 02:01:19.877589 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.877558    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/882bac36-975d-43b0-997d-e32a11b91009-maas-api-tls\") pod \"maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll\" (UID: \"882bac36-975d-43b0-997d-e32a11b91009\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"
Nov 03 02:01:19.949505 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:19.949468    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"
Nov 03 02:01:20.310042 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:20.310004    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79" event={"ID":"f10d4192-7fdc-4477-af07-2e37036a6595","Type":"ContainerStarted","Data":"6ac11694535c6b11a7af18d2ad37351b2bdc77fbc12d3a29ccc356c08c6a4fe2"}
Nov 03 02:01:20.310238 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:20.310070    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79" event={"ID":"f10d4192-7fdc-4477-af07-2e37036a6595","Type":"ContainerStarted","Data":"07edb1b71ce599393af28a94580a32241933865e8ead31c5e50ff67e49c7f4df"}
Nov 03 02:01:20.310293 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:20.310245    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79"
Nov 03 02:01:20.378529 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:01:20.378487    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882bac36_975d_43b0_997d_e32a11b91009.slice/crio-dc356794614bc102f5f77cf534791b4592cdf6d3cea8ad29eee806858ee04d60 WatchSource:0}: Error finding container dc356794614bc102f5f77cf534791b4592cdf6d3cea8ad29eee806858ee04d60: Status 404 returned error can't find the container with id dc356794614bc102f5f77cf534791b4592cdf6d3cea8ad29eee806858ee04d60
Nov 03 02:01:20.407871 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:20.407836    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"]
Nov 03 02:01:20.419492 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:20.419440    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79" podStartSLOduration=2.419426515 podStartE2EDuration="2.419426515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:20.417636664 +0000 UTC m=+607.431878030" watchObservedRunningTime="2026-11-03 02:01:20.419426515 +0000 UTC m=+607.433667880"
Nov 03 02:01:21.317398 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:21.317355    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll" event={"ID":"882bac36-975d-43b0-997d-e32a11b91009","Type":"ContainerStarted","Data":"dc356794614bc102f5f77cf534791b4592cdf6d3cea8ad29eee806858ee04d60"}
Nov 03 02:01:22.324384 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:22.324352    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll" event={"ID":"882bac36-975d-43b0-997d-e32a11b91009","Type":"ContainerStarted","Data":"e0cea3de74eb9e11245bb73626a8335e462bc059595cc04a4d7007ee12af9ea8"}
Nov 03 02:01:22.324909 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:22.324414    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"
Nov 03 02:01:22.433386 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:22.433325    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll" podStartSLOduration=3.39933568 podStartE2EDuration="4.433309484s" totalImagesPullingTime="1.033973804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:22.431416603 +0000 UTC m=+609.445657970" watchObservedRunningTime="2026-11-03 02:01:22.433309484 +0000 UTC m=+609.447550849"
Nov 03 02:01:23.226388 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:23.226360    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm"
Nov 03 02:01:23.268286 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:23.268249    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"
Nov 03 02:01:23.675324 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:23.675275    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"]
Nov 03 02:01:23.679635 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:23.679605    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"
Nov 03 02:01:23.712759 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:23.712680    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6234a2b0-8d37-4048-ab0b-27029c078d1d-openshift-service-ca\") pod \"maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7\" (UID: \"6234a2b0-8d37-4048-ab0b-27029c078d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"
Nov 03 02:01:23.712968 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:23.712849    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6234a2b0-8d37-4048-ab0b-27029c078d1d-maas-api-tls\") pod \"maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7\" (UID: \"6234a2b0-8d37-4048-ab0b-27029c078d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"
Nov 03 02:01:23.712968 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:23.712899    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6234a2b0-8d37-4048-ab0b-27029c078d1d-metrics-certs\") pod \"maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7\" (UID: \"6234a2b0-8d37-4048-ab0b-27029c078d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"
Nov 03 02:01:23.713057 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:23.713011    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5q8n\" (UniqueName: \"kubernetes.io/projected/6234a2b0-8d37-4048-ab0b-27029c078d1d-kube-api-access-b5q8n\") pod \"maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7\" (UID: \"6234a2b0-8d37-4048-ab0b-27029c078d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"
Nov 03 02:01:23.729902 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:23.729871    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-aead6b17-serving-cert\""
Nov 03 02:01:23.733872 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:23.733838    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"]
Nov 03 02:01:23.813888 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:23.813851    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6234a2b0-8d37-4048-ab0b-27029c078d1d-maas-api-tls\") pod \"maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7\" (UID: \"6234a2b0-8d37-4048-ab0b-27029c078d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"
Nov 03 02:01:23.814059 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:23.813903    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6234a2b0-8d37-4048-ab0b-27029c078d1d-metrics-certs\") pod \"maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7\" (UID: \"6234a2b0-8d37-4048-ab0b-27029c078d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"
Nov 03 02:01:23.814059 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:23.813956    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5q8n\" (UniqueName: \"kubernetes.io/projected/6234a2b0-8d37-4048-ab0b-27029c078d1d-kube-api-access-b5q8n\") pod \"maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7\" (UID: \"6234a2b0-8d37-4048-ab0b-27029c078d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"
Nov 03 02:01:23.814059 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:23.814003    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6234a2b0-8d37-4048-ab0b-27029c078d1d-openshift-service-ca\") pod \"maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7\" (UID: \"6234a2b0-8d37-4048-ab0b-27029c078d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"
Nov 03 02:01:23.814059 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:01:23.814015    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-serving-cert: secret "maas-api-e2e-ait-aead6b17-serving-cert" not found
Nov 03 02:01:23.814287 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:01:23.814090    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6234a2b0-8d37-4048-ab0b-27029c078d1d-maas-api-tls podName:6234a2b0-8d37-4048-ab0b-27029c078d1d nodeName:}" failed. No retries permitted until 2026-11-03 02:01:24.314071452 +0000 UTC m=+611.328312816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6234a2b0-8d37-4048-ab0b-27029c078d1d-maas-api-tls") pod "maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7" (UID: "6234a2b0-8d37-4048-ab0b-27029c078d1d") : secret "maas-api-e2e-ait-aead6b17-serving-cert" not found
Nov 03 02:01:23.814845 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:23.814810    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6234a2b0-8d37-4048-ab0b-27029c078d1d-openshift-service-ca\") pod \"maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7\" (UID: \"6234a2b0-8d37-4048-ab0b-27029c078d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"
Nov 03 02:01:23.817052 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:23.817021    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6234a2b0-8d37-4048-ab0b-27029c078d1d-metrics-certs\") pod \"maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7\" (UID: \"6234a2b0-8d37-4048-ab0b-27029c078d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"
Nov 03 02:01:23.867903 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:23.867872    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5q8n\" (UniqueName: \"kubernetes.io/projected/6234a2b0-8d37-4048-ab0b-27029c078d1d-kube-api-access-b5q8n\") pod \"maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7\" (UID: \"6234a2b0-8d37-4048-ab0b-27029c078d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"
Nov 03 02:01:24.318705 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:24.318664    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6234a2b0-8d37-4048-ab0b-27029c078d1d-maas-api-tls\") pod \"maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7\" (UID: \"6234a2b0-8d37-4048-ab0b-27029c078d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"
Nov 03 02:01:24.321142 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:24.321117    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6234a2b0-8d37-4048-ab0b-27029c078d1d-maas-api-tls\") pod \"maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7\" (UID: \"6234a2b0-8d37-4048-ab0b-27029c078d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"
Nov 03 02:01:24.593739 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:24.593621    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"
Nov 03 02:01:24.972024 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:24.971940    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"]
Nov 03 02:01:24.978678 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:01:24.978642    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6234a2b0_8d37_4048_ab0b_27029c078d1d.slice/crio-58402a9c9568c9bf7c22b861cf2a5dce28b2940430ff385168eb58f8ffd60511 WatchSource:0}: Error finding container 58402a9c9568c9bf7c22b861cf2a5dce28b2940430ff385168eb58f8ffd60511: Status 404 returned error can't find the container with id 58402a9c9568c9bf7c22b861cf2a5dce28b2940430ff385168eb58f8ffd60511
Nov 03 02:01:25.238506 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:25.238418    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz"
Nov 03 02:01:25.341390 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:25.341359    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7" event={"ID":"6234a2b0-8d37-4048-ab0b-27029c078d1d","Type":"ContainerStarted","Data":"58402a9c9568c9bf7c22b861cf2a5dce28b2940430ff385168eb58f8ffd60511"}
Nov 03 02:01:26.348081 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:26.348040    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7" event={"ID":"6234a2b0-8d37-4048-ab0b-27029c078d1d","Type":"ContainerStarted","Data":"076f0b7756f44c023c4f36771ed593f67f83861bcc927e1aff328d9f0458192a"}
Nov 03 02:01:26.348465 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:26.348138    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"
Nov 03 02:01:26.459153 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:26.459102    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7" podStartSLOduration=2.542786828 podStartE2EDuration="3.45908746s" totalImagesPullingTime="916.300632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:26.456041489 +0000 UTC m=+613.470282866" watchObservedRunningTime="2026-11-03 02:01:26.45908746 +0000 UTC m=+613.473328826"
Nov 03 02:01:28.265305 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:28.265272    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2"
Nov 03 02:01:28.265686 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:28.265513    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp"
Nov 03 02:01:28.336294 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:28.336268    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"
Nov 03 02:01:30.313191 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:30.313162    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9"
Nov 03 02:01:30.717443 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:01:30.717349    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:01:31.321311 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:31.321279    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79"
Nov 03 02:01:32.359967 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:32.359933    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"
Nov 03 02:01:33.359845 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:33.359805    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"]
Nov 03 02:01:33.364480 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:33.364453    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"
Nov 03 02:01:33.414350 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:33.414309    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-fbf24ed8-serving-cert\""
Nov 03 02:01:33.418041 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:33.417997    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wllgl\" (UniqueName: \"kubernetes.io/projected/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-kube-api-access-wllgl\") pod \"maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv\" (UID: \"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"
Nov 03 02:01:33.418209 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:33.418056    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-maas-api-tls\") pod \"maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv\" (UID: \"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"
Nov 03 02:01:33.418209 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:33.418090    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-metrics-certs\") pod \"maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv\" (UID: \"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"
Nov 03 02:01:33.418314 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:33.418221    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-openshift-service-ca\") pod \"maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv\" (UID: \"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"
Nov 03 02:01:33.463492 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:33.463449    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"]
Nov 03 02:01:33.519680 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:33.519644    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-openshift-service-ca\") pod \"maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv\" (UID: \"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"
Nov 03 02:01:33.519923 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:33.519763    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wllgl\" (UniqueName: \"kubernetes.io/projected/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-kube-api-access-wllgl\") pod \"maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv\" (UID: \"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"
Nov 03 02:01:33.519923 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:33.519795    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-maas-api-tls\") pod \"maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv\" (UID: \"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"
Nov 03 02:01:33.519923 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:33.519830    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-metrics-certs\") pod \"maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv\" (UID: \"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"
Nov 03 02:01:33.520116 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:01:33.519931    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-serving-cert: secret "maas-api-e2e-derive-fbf24ed8-serving-cert" not found
Nov 03 02:01:33.520116 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:01:33.520003    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-maas-api-tls podName:f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc nodeName:}" failed. No retries permitted until 2026-11-03 02:01:34.019983529 +0000 UTC m=+621.034224880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-maas-api-tls") pod "maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv" (UID: "f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc") : secret "maas-api-e2e-derive-fbf24ed8-serving-cert" not found
Nov 03 02:01:33.520573 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:33.520546    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-openshift-service-ca\") pod \"maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv\" (UID: \"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"
Nov 03 02:01:33.522573 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:33.522542    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-metrics-certs\") pod \"maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv\" (UID: \"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"
Nov 03 02:01:33.573983 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:33.573945    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wllgl\" (UniqueName: \"kubernetes.io/projected/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-kube-api-access-wllgl\") pod \"maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv\" (UID: \"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"
Nov 03 02:01:34.025476 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:34.025443    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-maas-api-tls\") pod \"maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv\" (UID: \"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"
Nov 03 02:01:34.027998 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:34.027975    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-maas-api-tls\") pod \"maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv\" (UID: \"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"
Nov 03 02:01:34.280946 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:34.280858    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"
Nov 03 02:01:34.667986 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:34.667901    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"]
Nov 03 02:01:34.675441 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:01:34.675406    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ed7bcd_76f1_45a6_8d34_a62e9eae70cc.slice/crio-52f911af679e82b51aeb0b9a85d2951f7b7ebf132106858b262d0bcd4c9a6dc3 WatchSource:0}: Error finding container 52f911af679e82b51aeb0b9a85d2951f7b7ebf132106858b262d0bcd4c9a6dc3: Status 404 returned error can't find the container with id 52f911af679e82b51aeb0b9a85d2951f7b7ebf132106858b262d0bcd4c9a6dc3
Nov 03 02:01:34.677011 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:34.676984    5005 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 02:01:35.404880 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:35.404848    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv" event={"ID":"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc","Type":"ContainerStarted","Data":"52f911af679e82b51aeb0b9a85d2951f7b7ebf132106858b262d0bcd4c9a6dc3"}
Nov 03 02:01:36.415096 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:36.415041    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv" event={"ID":"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc","Type":"ContainerStarted","Data":"342612a2affb16a164dd139e541d602329476ed595d07d0c451d9237a0e51542"}
Nov 03 02:01:36.415564 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:36.415262    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"
Nov 03 02:01:36.523030 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:36.522956    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv" podStartSLOduration=2.587150046 podStartE2EDuration="3.522940301s" totalImagesPullingTime="935.790255ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:36.521494667 +0000 UTC m=+623.535736034" watchObservedRunningTime="2026-11-03 02:01:36.522940301 +0000 UTC m=+623.537181667"
Nov 03 02:01:42.428021 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:42.427986    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"
Nov 03 02:01:42.968848 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:42.968803    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"]
Nov 03 02:01:42.974248 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:42.974223    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"
Nov 03 02:01:43.023822 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:43.023793    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-080015ba-serving-cert\""
Nov 03 02:01:43.027458 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:43.027427    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"]
Nov 03 02:01:43.118408 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:43.118363    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6sbd\" (UniqueName: \"kubernetes.io/projected/0fb8539d-0d8e-41a6-867d-ae58573b5e76-kube-api-access-h6sbd\") pod \"maas-api-e2e-ait-080015ba-7665dd8687-8k69l\" (UID: \"0fb8539d-0d8e-41a6-867d-ae58573b5e76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"
Nov 03 02:01:43.118624 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:43.118564    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fb8539d-0d8e-41a6-867d-ae58573b5e76-metrics-certs\") pod \"maas-api-e2e-ait-080015ba-7665dd8687-8k69l\" (UID: \"0fb8539d-0d8e-41a6-867d-ae58573b5e76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"
Nov 03 02:01:43.118727 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:43.118654    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb8539d-0d8e-41a6-867d-ae58573b5e76-openshift-service-ca\") pod \"maas-api-e2e-ait-080015ba-7665dd8687-8k69l\" (UID: \"0fb8539d-0d8e-41a6-867d-ae58573b5e76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"
Nov 03 02:01:43.118795 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:43.118734    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fb8539d-0d8e-41a6-867d-ae58573b5e76-maas-api-tls\") pod \"maas-api-e2e-ait-080015ba-7665dd8687-8k69l\" (UID: \"0fb8539d-0d8e-41a6-867d-ae58573b5e76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"
Nov 03 02:01:43.219578 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:43.219477    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fb8539d-0d8e-41a6-867d-ae58573b5e76-metrics-certs\") pod \"maas-api-e2e-ait-080015ba-7665dd8687-8k69l\" (UID: \"0fb8539d-0d8e-41a6-867d-ae58573b5e76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"
Nov 03 02:01:43.219823 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:43.219584    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb8539d-0d8e-41a6-867d-ae58573b5e76-openshift-service-ca\") pod \"maas-api-e2e-ait-080015ba-7665dd8687-8k69l\" (UID: \"0fb8539d-0d8e-41a6-867d-ae58573b5e76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"
Nov 03 02:01:43.219823 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:43.219715    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fb8539d-0d8e-41a6-867d-ae58573b5e76-maas-api-tls\") pod \"maas-api-e2e-ait-080015ba-7665dd8687-8k69l\" (UID: \"0fb8539d-0d8e-41a6-867d-ae58573b5e76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"
Nov 03 02:01:43.219823 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:43.219769    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6sbd\" (UniqueName: \"kubernetes.io/projected/0fb8539d-0d8e-41a6-867d-ae58573b5e76-kube-api-access-h6sbd\") pod \"maas-api-e2e-ait-080015ba-7665dd8687-8k69l\" (UID: \"0fb8539d-0d8e-41a6-867d-ae58573b5e76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"
Nov 03 02:01:43.220826 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:43.220790    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb8539d-0d8e-41a6-867d-ae58573b5e76-openshift-service-ca\") pod \"maas-api-e2e-ait-080015ba-7665dd8687-8k69l\" (UID: \"0fb8539d-0d8e-41a6-867d-ae58573b5e76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"
Nov 03 02:01:43.223206 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:43.223182    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fb8539d-0d8e-41a6-867d-ae58573b5e76-maas-api-tls\") pod \"maas-api-e2e-ait-080015ba-7665dd8687-8k69l\" (UID: \"0fb8539d-0d8e-41a6-867d-ae58573b5e76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"
Nov 03 02:01:43.224845 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:43.224780    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fb8539d-0d8e-41a6-867d-ae58573b5e76-metrics-certs\") pod \"maas-api-e2e-ait-080015ba-7665dd8687-8k69l\" (UID: \"0fb8539d-0d8e-41a6-867d-ae58573b5e76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"
Nov 03 02:01:43.275489 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:43.275455    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6sbd\" (UniqueName: \"kubernetes.io/projected/0fb8539d-0d8e-41a6-867d-ae58573b5e76-kube-api-access-h6sbd\") pod \"maas-api-e2e-ait-080015ba-7665dd8687-8k69l\" (UID: \"0fb8539d-0d8e-41a6-867d-ae58573b5e76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"
Nov 03 02:01:43.288322 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:43.288295    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"
Nov 03 02:01:43.660061 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:43.660018    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"]
Nov 03 02:01:43.666438 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:01:43.666406    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb8539d_0d8e_41a6_867d_ae58573b5e76.slice/crio-eb7eb58098da82fc6b43e1aaf699767d29e5c1e75a4fbebc8dff0c2b7510adbc WatchSource:0}: Error finding container eb7eb58098da82fc6b43e1aaf699767d29e5c1e75a4fbebc8dff0c2b7510adbc: Status 404 returned error can't find the container with id eb7eb58098da82fc6b43e1aaf699767d29e5c1e75a4fbebc8dff0c2b7510adbc
Nov 03 02:01:44.462439 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:44.462399    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l" event={"ID":"0fb8539d-0d8e-41a6-867d-ae58573b5e76","Type":"ContainerStarted","Data":"eb7eb58098da82fc6b43e1aaf699767d29e5c1e75a4fbebc8dff0c2b7510adbc"}
Nov 03 02:01:45.469617 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:45.469583    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l" event={"ID":"0fb8539d-0d8e-41a6-867d-ae58573b5e76","Type":"ContainerStarted","Data":"caeeea65576e6a9c1344ae4bb34a74da601c2d2469cf8bd35e92432f076eaeb7"}
Nov 03 02:01:45.470050 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:45.469668    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"
Nov 03 02:01:45.576827 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:45.576622    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l" podStartSLOduration=2.423560716 podStartE2EDuration="3.576607486s" totalImagesPullingTime="1.15304677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:45.575893999 +0000 UTC m=+632.590135365" watchObservedRunningTime="2026-11-03 02:01:45.576607486 +0000 UTC m=+632.590848853"
Nov 03 02:01:51.481147 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:01:51.481116    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"
Nov 03 02:02:00.206709 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:00.206658    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2"]
Nov 03 02:02:00.312579 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:00.312549    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp"]
Nov 03 02:02:00.711757 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:02:00.711720    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:02:05.207223 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.207167    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2" podUID="04b6c270-905c-467d-b07d-6d22ebf79a7b" containerName="payload-processing" containerID="cri-o://6fe186a005068d8bf04ee723b20ad884ed2989e984a2a8d7c4923b648c5437e5" gracePeriod=25
Nov 03 02:02:05.313204 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.313169    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp" podUID="0275219a-ec56-447b-9b83-19ec8008a85f" containerName="payload-pre-processing" containerID="cri-o://37d3b5c9b4d40ab894aa09d1d2d7ec2f14403a4eef3b9c108c2a5ae131b03a30" gracePeriod=25
Nov 03 02:02:05.465633 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.465569    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2"
Nov 03 02:02:05.545610 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.545584    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04b6c270-905c-467d-b07d-6d22ebf79a7b-kube-api-access-qwqb7\" (UniqueName: \"kubernetes.io/projected/04b6c270-905c-467d-b07d-6d22ebf79a7b-kube-api-access-qwqb7\") pod \"04b6c270-905c-467d-b07d-6d22ebf79a7b\" (UID: \"04b6c270-905c-467d-b07d-6d22ebf79a7b\") "
Nov 03 02:02:05.545779 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.545670    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04b6c270-905c-467d-b07d-6d22ebf79a7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b6c270-905c-467d-b07d-6d22ebf79a7b-plugins-config-volume\") pod \"04b6c270-905c-467d-b07d-6d22ebf79a7b\" (UID: \"04b6c270-905c-467d-b07d-6d22ebf79a7b\") "
Nov 03 02:02:05.546107 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.546045    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b6c270-905c-467d-b07d-6d22ebf79a7b-plugins-config-volume" pod "04b6c270-905c-467d-b07d-6d22ebf79a7b" (UID: "04b6c270-905c-467d-b07d-6d22ebf79a7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:05.546177 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.546159    5005 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b6c270-905c-467d-b07d-6d22ebf79a7b-plugins-config-volume\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:05.547966 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.547944    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b6c270-905c-467d-b07d-6d22ebf79a7b-kube-api-access-qwqb7" pod "04b6c270-905c-467d-b07d-6d22ebf79a7b" (UID: "04b6c270-905c-467d-b07d-6d22ebf79a7b"). InnerVolumeSpecName "kube-api-access-qwqb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:05.565749 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.565731    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp"
Nov 03 02:02:05.585419 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.585395    5005 generic.go:356] "Generic (PLEG): container finished" podID="0275219a-ec56-447b-9b83-19ec8008a85f" containerID="37d3b5c9b4d40ab894aa09d1d2d7ec2f14403a4eef3b9c108c2a5ae131b03a30" exitCode=0
Nov 03 02:02:05.585517 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.585433    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp"
Nov 03 02:02:05.585517 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.585477    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp" event={"ID":"0275219a-ec56-447b-9b83-19ec8008a85f","Type":"ContainerDied","Data":"37d3b5c9b4d40ab894aa09d1d2d7ec2f14403a4eef3b9c108c2a5ae131b03a30"}
Nov 03 02:02:05.585628 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.585558    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp" event={"ID":"0275219a-ec56-447b-9b83-19ec8008a85f","Type":"ContainerDied","Data":"9623b86ac9b4bcacc72a62e8dbeeba7e476f19d22abdf468f828960a92f3f2fe"}
Nov 03 02:02:05.585628 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.585588    5005 scope.go:122] "RemoveContainer" containerID="37d3b5c9b4d40ab894aa09d1d2d7ec2f14403a4eef3b9c108c2a5ae131b03a30"
Nov 03 02:02:05.586816 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.586793    5005 generic.go:356] "Generic (PLEG): container finished" podID="04b6c270-905c-467d-b07d-6d22ebf79a7b" containerID="6fe186a005068d8bf04ee723b20ad884ed2989e984a2a8d7c4923b648c5437e5" exitCode=0
Nov 03 02:02:05.586915 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.586825    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2" event={"ID":"04b6c270-905c-467d-b07d-6d22ebf79a7b","Type":"ContainerDied","Data":"6fe186a005068d8bf04ee723b20ad884ed2989e984a2a8d7c4923b648c5437e5"}
Nov 03 02:02:05.586915 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.586853    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2"
Nov 03 02:02:05.586915 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.586876    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2" event={"ID":"04b6c270-905c-467d-b07d-6d22ebf79a7b","Type":"ContainerDied","Data":"19b32431bdb73ab258b36a5d5c9d6239782f5b5f4b2bb12c1d6f7ee64707d274"}
Nov 03 02:02:05.595188 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.595173    5005 scope.go:122] "RemoveContainer" containerID="37d3b5c9b4d40ab894aa09d1d2d7ec2f14403a4eef3b9c108c2a5ae131b03a30"
Nov 03 02:02:05.595444 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:02:05.595426    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d3b5c9b4d40ab894aa09d1d2d7ec2f14403a4eef3b9c108c2a5ae131b03a30\": container with ID starting with 37d3b5c9b4d40ab894aa09d1d2d7ec2f14403a4eef3b9c108c2a5ae131b03a30 not found: ID does not exist" containerID="37d3b5c9b4d40ab894aa09d1d2d7ec2f14403a4eef3b9c108c2a5ae131b03a30"
Nov 03 02:02:05.595509 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.595453    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d3b5c9b4d40ab894aa09d1d2d7ec2f14403a4eef3b9c108c2a5ae131b03a30"} err="failed to get container status \"37d3b5c9b4d40ab894aa09d1d2d7ec2f14403a4eef3b9c108c2a5ae131b03a30\": rpc error: code = NotFound desc = could not find container \"37d3b5c9b4d40ab894aa09d1d2d7ec2f14403a4eef3b9c108c2a5ae131b03a30\": container with ID starting with 37d3b5c9b4d40ab894aa09d1d2d7ec2f14403a4eef3b9c108c2a5ae131b03a30 not found: ID does not exist"
Nov 03 02:02:05.595509 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.595469    5005 scope.go:122] "RemoveContainer" containerID="6fe186a005068d8bf04ee723b20ad884ed2989e984a2a8d7c4923b648c5437e5"
Nov 03 02:02:05.605782 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.605760    5005 scope.go:122] "RemoveContainer" containerID="6fe186a005068d8bf04ee723b20ad884ed2989e984a2a8d7c4923b648c5437e5"
Nov 03 02:02:05.606036 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:02:05.606016    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fe186a005068d8bf04ee723b20ad884ed2989e984a2a8d7c4923b648c5437e5\": container with ID starting with 6fe186a005068d8bf04ee723b20ad884ed2989e984a2a8d7c4923b648c5437e5 not found: ID does not exist" containerID="6fe186a005068d8bf04ee723b20ad884ed2989e984a2a8d7c4923b648c5437e5"
Nov 03 02:02:05.606095 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.606041    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe186a005068d8bf04ee723b20ad884ed2989e984a2a8d7c4923b648c5437e5"} err="failed to get container status \"6fe186a005068d8bf04ee723b20ad884ed2989e984a2a8d7c4923b648c5437e5\": rpc error: code = NotFound desc = could not find container \"6fe186a005068d8bf04ee723b20ad884ed2989e984a2a8d7c4923b648c5437e5\": container with ID starting with 6fe186a005068d8bf04ee723b20ad884ed2989e984a2a8d7c4923b648c5437e5 not found: ID does not exist"
Nov 03 02:02:05.647462 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.647440    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0275219a-ec56-447b-9b83-19ec8008a85f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0275219a-ec56-447b-9b83-19ec8008a85f-plugins-config-volume\") pod \"0275219a-ec56-447b-9b83-19ec8008a85f\" (UID: \"0275219a-ec56-447b-9b83-19ec8008a85f\") "
Nov 03 02:02:05.647546 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.647484    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0275219a-ec56-447b-9b83-19ec8008a85f-kube-api-access-8bjf6\" (UniqueName: \"kubernetes.io/projected/0275219a-ec56-447b-9b83-19ec8008a85f-kube-api-access-8bjf6\") pod \"0275219a-ec56-447b-9b83-19ec8008a85f\" (UID: \"0275219a-ec56-447b-9b83-19ec8008a85f\") "
Nov 03 02:02:05.647759 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.647741    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0275219a-ec56-447b-9b83-19ec8008a85f-plugins-config-volume" pod "0275219a-ec56-447b-9b83-19ec8008a85f" (UID: "0275219a-ec56-447b-9b83-19ec8008a85f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:05.647811 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.647748    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwqb7\" (UniqueName: \"kubernetes.io/projected/04b6c270-905c-467d-b07d-6d22ebf79a7b-kube-api-access-qwqb7\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:05.649315 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.649293    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0275219a-ec56-447b-9b83-19ec8008a85f-kube-api-access-8bjf6" pod "0275219a-ec56-447b-9b83-19ec8008a85f" (UID: "0275219a-ec56-447b-9b83-19ec8008a85f"). InnerVolumeSpecName "kube-api-access-8bjf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:05.748457 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.748397    5005 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0275219a-ec56-447b-9b83-19ec8008a85f-plugins-config-volume\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:05.748457 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.748418    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bjf6\" (UniqueName: \"kubernetes.io/projected/0275219a-ec56-447b-9b83-19ec8008a85f-kube-api-access-8bjf6\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:05.780537 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.780515    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2"]
Nov 03 02:02:05.783639 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.783618    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6ac43c49-79f67478bf-pscn2"]
Nov 03 02:02:05.997897 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:05.997875    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp"]
Nov 03 02:02:06.001335 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:06.001287    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6ac43c49-59564d5bdc-88rdp"]
Nov 03 02:02:07.109930 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:07.109898    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0275219a-ec56-447b-9b83-19ec8008a85f" path="/var/lib/kubelet/pods/0275219a-ec56-447b-9b83-19ec8008a85f/volumes"
Nov 03 02:02:07.110318 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:07.110272    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04b6c270-905c-467d-b07d-6d22ebf79a7b" path="/var/lib/kubelet/pods/04b6c270-905c-467d-b07d-6d22ebf79a7b/volumes"
Nov 03 02:02:07.973245 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:07.973188    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"]
Nov 03 02:02:07.974009 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:07.973976    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82" podUID="89afc39a-86cd-46e9-bb50-bd3a3e2d0007" containerName="maas-api" containerID="cri-o://de576f33d899a2c84f2095f867e0e155ae2a9ad7cb4c7370657bf27854c409c1" gracePeriod=30
Nov 03 02:02:08.243628 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:08.243600    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"
Nov 03 02:02:08.272586 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:08.272566    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-openshift-service-ca\") pod \"89afc39a-86cd-46e9-bb50-bd3a3e2d0007\" (UID: \"89afc39a-86cd-46e9-bb50-bd3a3e2d0007\") "
Nov 03 02:02:08.272735 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:08.272653    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-kube-api-access-brqjv\" (UniqueName: \"kubernetes.io/projected/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-kube-api-access-brqjv\") pod \"89afc39a-86cd-46e9-bb50-bd3a3e2d0007\" (UID: \"89afc39a-86cd-46e9-bb50-bd3a3e2d0007\") "
Nov 03 02:02:08.272791 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:08.272774    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-metrics-certs\") pod \"89afc39a-86cd-46e9-bb50-bd3a3e2d0007\" (UID: \"89afc39a-86cd-46e9-bb50-bd3a3e2d0007\") "
Nov 03 02:02:08.272848 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:08.272832    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-maas-api-tls\") pod \"89afc39a-86cd-46e9-bb50-bd3a3e2d0007\" (UID: \"89afc39a-86cd-46e9-bb50-bd3a3e2d0007\") "
Nov 03 02:02:08.273013 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:08.272977    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-openshift-service-ca" pod "89afc39a-86cd-46e9-bb50-bd3a3e2d0007" (UID: "89afc39a-86cd-46e9-bb50-bd3a3e2d0007"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:08.273237 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:08.273218    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:08.274802 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:08.274767    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-kube-api-access-brqjv" pod "89afc39a-86cd-46e9-bb50-bd3a3e2d0007" (UID: "89afc39a-86cd-46e9-bb50-bd3a3e2d0007"). InnerVolumeSpecName "kube-api-access-brqjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:08.274914 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:08.274805    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-maas-api-tls" pod "89afc39a-86cd-46e9-bb50-bd3a3e2d0007" (UID: "89afc39a-86cd-46e9-bb50-bd3a3e2d0007"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:08.274914 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:08.274860    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-metrics-certs" pod "89afc39a-86cd-46e9-bb50-bd3a3e2d0007" (UID: "89afc39a-86cd-46e9-bb50-bd3a3e2d0007"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:08.374147 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:08.374119    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brqjv\" (UniqueName: \"kubernetes.io/projected/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-kube-api-access-brqjv\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:08.374147 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:08.374145    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:08.374298 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:08.374155    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89afc39a-86cd-46e9-bb50-bd3a3e2d0007-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:08.608211 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:08.608177    5005 generic.go:356] "Generic (PLEG): container finished" podID="89afc39a-86cd-46e9-bb50-bd3a3e2d0007" containerID="de576f33d899a2c84f2095f867e0e155ae2a9ad7cb4c7370657bf27854c409c1" exitCode=0
Nov 03 02:02:08.608390 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:08.608235    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"
Nov 03 02:02:08.608390 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:08.608257    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82" event={"ID":"89afc39a-86cd-46e9-bb50-bd3a3e2d0007","Type":"ContainerDied","Data":"de576f33d899a2c84f2095f867e0e155ae2a9ad7cb4c7370657bf27854c409c1"}
Nov 03 02:02:08.608390 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:08.608342    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82" event={"ID":"89afc39a-86cd-46e9-bb50-bd3a3e2d0007","Type":"ContainerDied","Data":"5611b0e6ebfb7430563d2eebeea9336e06127d9a586e1c9f728a9fd41bc26f47"}
Nov 03 02:02:08.608390 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:08.608371    5005 scope.go:122] "RemoveContainer" containerID="de576f33d899a2c84f2095f867e0e155ae2a9ad7cb4c7370657bf27854c409c1"
Nov 03 02:02:08.618403 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:08.618387    5005 scope.go:122] "RemoveContainer" containerID="de576f33d899a2c84f2095f867e0e155ae2a9ad7cb4c7370657bf27854c409c1"
Nov 03 02:02:08.618661 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:02:08.618639    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de576f33d899a2c84f2095f867e0e155ae2a9ad7cb4c7370657bf27854c409c1\": container with ID starting with de576f33d899a2c84f2095f867e0e155ae2a9ad7cb4c7370657bf27854c409c1 not found: ID does not exist" containerID="de576f33d899a2c84f2095f867e0e155ae2a9ad7cb4c7370657bf27854c409c1"
Nov 03 02:02:08.618769 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:08.618667    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de576f33d899a2c84f2095f867e0e155ae2a9ad7cb4c7370657bf27854c409c1"} err="failed to get container status \"de576f33d899a2c84f2095f867e0e155ae2a9ad7cb4c7370657bf27854c409c1\": rpc error: code = NotFound desc = could not find container \"de576f33d899a2c84f2095f867e0e155ae2a9ad7cb4c7370657bf27854c409c1\": container with ID starting with de576f33d899a2c84f2095f867e0e155ae2a9ad7cb4c7370657bf27854c409c1 not found: ID does not exist"
Nov 03 02:02:08.761426 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:08.761405    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"]
Nov 03 02:02:08.807604 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:08.807579    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ac43c49-55df997955-nmn82"]
Nov 03 02:02:09.108602 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:09.108573    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89afc39a-86cd-46e9-bb50-bd3a3e2d0007" path="/var/lib/kubelet/pods/89afc39a-86cd-46e9-bb50-bd3a3e2d0007/volumes"
Nov 03 02:02:14.209046 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.209006    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"]
Nov 03 02:02:14.209611 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.209215    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll" podUID="882bac36-975d-43b0-997d-e32a11b91009" containerName="maas-api" containerID="cri-o://e0cea3de74eb9e11245bb73626a8335e462bc059595cc04a4d7007ee12af9ea8" gracePeriod=30
Nov 03 02:02:14.467528 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.467468    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"
Nov 03 02:02:14.534472 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.534444    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/882bac36-975d-43b0-997d-e32a11b91009-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/882bac36-975d-43b0-997d-e32a11b91009-openshift-service-ca\") pod \"882bac36-975d-43b0-997d-e32a11b91009\" (UID: \"882bac36-975d-43b0-997d-e32a11b91009\") "
Nov 03 02:02:14.534627 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.534480    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/882bac36-975d-43b0-997d-e32a11b91009-metrics-certs\" (UniqueName: \"kubernetes.io/secret/882bac36-975d-43b0-997d-e32a11b91009-metrics-certs\") pod \"882bac36-975d-43b0-997d-e32a11b91009\" (UID: \"882bac36-975d-43b0-997d-e32a11b91009\") "
Nov 03 02:02:14.534627 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.534523    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/882bac36-975d-43b0-997d-e32a11b91009-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/882bac36-975d-43b0-997d-e32a11b91009-maas-api-tls\") pod \"882bac36-975d-43b0-997d-e32a11b91009\" (UID: \"882bac36-975d-43b0-997d-e32a11b91009\") "
Nov 03 02:02:14.534627 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.534570    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/882bac36-975d-43b0-997d-e32a11b91009-kube-api-access-xg6ws\" (UniqueName: \"kubernetes.io/projected/882bac36-975d-43b0-997d-e32a11b91009-kube-api-access-xg6ws\") pod \"882bac36-975d-43b0-997d-e32a11b91009\" (UID: \"882bac36-975d-43b0-997d-e32a11b91009\") "
Nov 03 02:02:14.534799 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.534779    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/882bac36-975d-43b0-997d-e32a11b91009-openshift-service-ca" pod "882bac36-975d-43b0-997d-e32a11b91009" (UID: "882bac36-975d-43b0-997d-e32a11b91009"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:14.534998 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.534968    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/882bac36-975d-43b0-997d-e32a11b91009-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:14.536642 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.536613    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882bac36-975d-43b0-997d-e32a11b91009-kube-api-access-xg6ws" pod "882bac36-975d-43b0-997d-e32a11b91009" (UID: "882bac36-975d-43b0-997d-e32a11b91009"). InnerVolumeSpecName "kube-api-access-xg6ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:14.536642 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.536634    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/882bac36-975d-43b0-997d-e32a11b91009-metrics-certs" pod "882bac36-975d-43b0-997d-e32a11b91009" (UID: "882bac36-975d-43b0-997d-e32a11b91009"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:14.536823 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.536658    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/882bac36-975d-43b0-997d-e32a11b91009-maas-api-tls" pod "882bac36-975d-43b0-997d-e32a11b91009" (UID: "882bac36-975d-43b0-997d-e32a11b91009"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:14.636353 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.636328    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/882bac36-975d-43b0-997d-e32a11b91009-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:14.636353 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.636351    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg6ws\" (UniqueName: \"kubernetes.io/projected/882bac36-975d-43b0-997d-e32a11b91009-kube-api-access-xg6ws\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:14.636504 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.636362    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/882bac36-975d-43b0-997d-e32a11b91009-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:14.644309 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.644281    5005 generic.go:356] "Generic (PLEG): container finished" podID="882bac36-975d-43b0-997d-e32a11b91009" containerID="e0cea3de74eb9e11245bb73626a8335e462bc059595cc04a4d7007ee12af9ea8" exitCode=0
Nov 03 02:02:14.644425 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.644329    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"
Nov 03 02:02:14.644425 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.644366    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll" event={"ID":"882bac36-975d-43b0-997d-e32a11b91009","Type":"ContainerDied","Data":"e0cea3de74eb9e11245bb73626a8335e462bc059595cc04a4d7007ee12af9ea8"}
Nov 03 02:02:14.644425 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.644416    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll" event={"ID":"882bac36-975d-43b0-997d-e32a11b91009","Type":"ContainerDied","Data":"dc356794614bc102f5f77cf534791b4592cdf6d3cea8ad29eee806858ee04d60"}
Nov 03 02:02:14.644583 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.644437    5005 scope.go:122] "RemoveContainer" containerID="e0cea3de74eb9e11245bb73626a8335e462bc059595cc04a4d7007ee12af9ea8"
Nov 03 02:02:14.654428 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.654411    5005 scope.go:122] "RemoveContainer" containerID="e0cea3de74eb9e11245bb73626a8335e462bc059595cc04a4d7007ee12af9ea8"
Nov 03 02:02:14.654716 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:02:14.654682    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0cea3de74eb9e11245bb73626a8335e462bc059595cc04a4d7007ee12af9ea8\": container with ID starting with e0cea3de74eb9e11245bb73626a8335e462bc059595cc04a4d7007ee12af9ea8 not found: ID does not exist" containerID="e0cea3de74eb9e11245bb73626a8335e462bc059595cc04a4d7007ee12af9ea8"
Nov 03 02:02:14.654784 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.654723    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0cea3de74eb9e11245bb73626a8335e462bc059595cc04a4d7007ee12af9ea8"} err="failed to get container status \"e0cea3de74eb9e11245bb73626a8335e462bc059595cc04a4d7007ee12af9ea8\": rpc error: code = NotFound desc = could not find container \"e0cea3de74eb9e11245bb73626a8335e462bc059595cc04a4d7007ee12af9ea8\": container with ID starting with e0cea3de74eb9e11245bb73626a8335e462bc059595cc04a4d7007ee12af9ea8 not found: ID does not exist"
Nov 03 02:02:14.799887 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.799851    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"]
Nov 03 02:02:14.802821 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:14.802792    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7683f606-8d4fddc95-gmvll"]
Nov 03 02:02:15.108754 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:15.108668    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="882bac36-975d-43b0-997d-e32a11b91009" path="/var/lib/kubelet/pods/882bac36-975d-43b0-997d-e32a11b91009/volumes"
Nov 03 02:02:16.604729 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:16.604685    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm"]
Nov 03 02:02:16.793805 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:16.793773    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz"]
Nov 03 02:02:21.605226 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:21.605185    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm" podUID="20c97e3c-7ac2-49f5-9391-2ac5f64d8766" containerName="payload-processing" containerID="cri-o://acc8e85218924eb0f97164e786d650caa55dcccb9f4570f85d8871e5d348b5c7" gracePeriod=25
Nov 03 02:02:21.794245 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:21.794212    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz" podUID="33f9915e-45ca-4ab4-8466-0b4967abed94" containerName="payload-pre-processing" containerID="cri-o://665e8e4855874341e21984e753e05237989471702bd6776e18fbd1c4ebff5c87" gracePeriod=25
Nov 03 02:02:21.922594 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:21.922572    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm"
Nov 03 02:02:22.016868 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.015369    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20c97e3c-7ac2-49f5-9391-2ac5f64d8766-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c97e3c-7ac2-49f5-9391-2ac5f64d8766-plugins-config-volume\") pod \"20c97e3c-7ac2-49f5-9391-2ac5f64d8766\" (UID: \"20c97e3c-7ac2-49f5-9391-2ac5f64d8766\") "
Nov 03 02:02:22.016868 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.015620    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c97e3c-7ac2-49f5-9391-2ac5f64d8766-kube-api-access-8zr5z\" (UniqueName: \"kubernetes.io/projected/20c97e3c-7ac2-49f5-9391-2ac5f64d8766-kube-api-access-8zr5z\") pod \"20c97e3c-7ac2-49f5-9391-2ac5f64d8766\" (UID: \"20c97e3c-7ac2-49f5-9391-2ac5f64d8766\") "
Nov 03 02:02:22.019094 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.019063    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c97e3c-7ac2-49f5-9391-2ac5f64d8766-plugins-config-volume" pod "20c97e3c-7ac2-49f5-9391-2ac5f64d8766" (UID: "20c97e3c-7ac2-49f5-9391-2ac5f64d8766"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:22.019770 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.019749    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c97e3c-7ac2-49f5-9391-2ac5f64d8766-kube-api-access-8zr5z" pod "20c97e3c-7ac2-49f5-9391-2ac5f64d8766" (UID: "20c97e3c-7ac2-49f5-9391-2ac5f64d8766"). InnerVolumeSpecName "kube-api-access-8zr5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:22.048391 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.048366    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz"
Nov 03 02:02:22.116924 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.116897    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33f9915e-45ca-4ab4-8466-0b4967abed94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f9915e-45ca-4ab4-8466-0b4967abed94-plugins-config-volume\") pod \"33f9915e-45ca-4ab4-8466-0b4967abed94\" (UID: \"33f9915e-45ca-4ab4-8466-0b4967abed94\") "
Nov 03 02:02:22.117089 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.117012    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33f9915e-45ca-4ab4-8466-0b4967abed94-kube-api-access-shlzn\" (UniqueName: \"kubernetes.io/projected/33f9915e-45ca-4ab4-8466-0b4967abed94-kube-api-access-shlzn\") pod \"33f9915e-45ca-4ab4-8466-0b4967abed94\" (UID: \"33f9915e-45ca-4ab4-8466-0b4967abed94\") "
Nov 03 02:02:22.117226 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.117211    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zr5z\" (UniqueName: \"kubernetes.io/projected/20c97e3c-7ac2-49f5-9391-2ac5f64d8766-kube-api-access-8zr5z\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:22.117293 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.117228    5005 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c97e3c-7ac2-49f5-9391-2ac5f64d8766-plugins-config-volume\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:22.117338 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.117286    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33f9915e-45ca-4ab4-8466-0b4967abed94-plugins-config-volume" pod "33f9915e-45ca-4ab4-8466-0b4967abed94" (UID: "33f9915e-45ca-4ab4-8466-0b4967abed94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:22.119005 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.118984    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f9915e-45ca-4ab4-8466-0b4967abed94-kube-api-access-shlzn" pod "33f9915e-45ca-4ab4-8466-0b4967abed94" (UID: "33f9915e-45ca-4ab4-8466-0b4967abed94"). InnerVolumeSpecName "kube-api-access-shlzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:22.219467 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.218597    5005 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f9915e-45ca-4ab4-8466-0b4967abed94-plugins-config-volume\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:22.219467 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.218628    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shlzn\" (UniqueName: \"kubernetes.io/projected/33f9915e-45ca-4ab4-8466-0b4967abed94-kube-api-access-shlzn\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:22.691197 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.691165    5005 generic.go:356] "Generic (PLEG): container finished" podID="33f9915e-45ca-4ab4-8466-0b4967abed94" containerID="665e8e4855874341e21984e753e05237989471702bd6776e18fbd1c4ebff5c87" exitCode=0
Nov 03 02:02:22.691603 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.691210    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz"
Nov 03 02:02:22.691603 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.691251    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz" event={"ID":"33f9915e-45ca-4ab4-8466-0b4967abed94","Type":"ContainerDied","Data":"665e8e4855874341e21984e753e05237989471702bd6776e18fbd1c4ebff5c87"}
Nov 03 02:02:22.691603 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.691297    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz" event={"ID":"33f9915e-45ca-4ab4-8466-0b4967abed94","Type":"ContainerDied","Data":"be73f59dac12156b76ce0f0ef05244e3f2284e88d1c842bafc5bee15065b52af"}
Nov 03 02:02:22.691603 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.691313    5005 scope.go:122] "RemoveContainer" containerID="665e8e4855874341e21984e753e05237989471702bd6776e18fbd1c4ebff5c87"
Nov 03 02:02:22.692775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.692747    5005 generic.go:356] "Generic (PLEG): container finished" podID="20c97e3c-7ac2-49f5-9391-2ac5f64d8766" containerID="acc8e85218924eb0f97164e786d650caa55dcccb9f4570f85d8871e5d348b5c7" exitCode=0
Nov 03 02:02:22.692860 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.692784    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm"
Nov 03 02:02:22.692903 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.692782    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm" event={"ID":"20c97e3c-7ac2-49f5-9391-2ac5f64d8766","Type":"ContainerDied","Data":"acc8e85218924eb0f97164e786d650caa55dcccb9f4570f85d8871e5d348b5c7"}
Nov 03 02:02:22.692903 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.692895    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm" event={"ID":"20c97e3c-7ac2-49f5-9391-2ac5f64d8766","Type":"ContainerDied","Data":"e5fc882b4cf090bd8259221ba00ca3f0daae1565fafbd6a1b01129035533328d"}
Nov 03 02:02:22.701656 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.701637    5005 scope.go:122] "RemoveContainer" containerID="665e8e4855874341e21984e753e05237989471702bd6776e18fbd1c4ebff5c87"
Nov 03 02:02:22.702009 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:02:22.701984    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"665e8e4855874341e21984e753e05237989471702bd6776e18fbd1c4ebff5c87\": container with ID starting with 665e8e4855874341e21984e753e05237989471702bd6776e18fbd1c4ebff5c87 not found: ID does not exist" containerID="665e8e4855874341e21984e753e05237989471702bd6776e18fbd1c4ebff5c87"
Nov 03 02:02:22.702093 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.702017    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"665e8e4855874341e21984e753e05237989471702bd6776e18fbd1c4ebff5c87"} err="failed to get container status \"665e8e4855874341e21984e753e05237989471702bd6776e18fbd1c4ebff5c87\": rpc error: code = NotFound desc = could not find container \"665e8e4855874341e21984e753e05237989471702bd6776e18fbd1c4ebff5c87\": container with ID starting with 665e8e4855874341e21984e753e05237989471702bd6776e18fbd1c4ebff5c87 not found: ID does not exist"
Nov 03 02:02:22.702093 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.702034    5005 scope.go:122] "RemoveContainer" containerID="acc8e85218924eb0f97164e786d650caa55dcccb9f4570f85d8871e5d348b5c7"
Nov 03 02:02:22.710920 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.710905    5005 scope.go:122] "RemoveContainer" containerID="acc8e85218924eb0f97164e786d650caa55dcccb9f4570f85d8871e5d348b5c7"
Nov 03 02:02:22.711146 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:02:22.711127    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc8e85218924eb0f97164e786d650caa55dcccb9f4570f85d8871e5d348b5c7\": container with ID starting with acc8e85218924eb0f97164e786d650caa55dcccb9f4570f85d8871e5d348b5c7 not found: ID does not exist" containerID="acc8e85218924eb0f97164e786d650caa55dcccb9f4570f85d8871e5d348b5c7"
Nov 03 02:02:22.711203 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.711150    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc8e85218924eb0f97164e786d650caa55dcccb9f4570f85d8871e5d348b5c7"} err="failed to get container status \"acc8e85218924eb0f97164e786d650caa55dcccb9f4570f85d8871e5d348b5c7\": rpc error: code = NotFound desc = could not find container \"acc8e85218924eb0f97164e786d650caa55dcccb9f4570f85d8871e5d348b5c7\": container with ID starting with acc8e85218924eb0f97164e786d650caa55dcccb9f4570f85d8871e5d348b5c7 not found: ID does not exist"
Nov 03 02:02:22.850613 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.850553    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz"]
Nov 03 02:02:22.896976 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.896929    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aead6b17-7877c56b7f-jq9kz"]
Nov 03 02:02:22.959103 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.959080    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm"]
Nov 03 02:02:22.962500 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:22.962476    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aead6b17-66d5c4568f-lzkbm"]
Nov 03 02:02:23.108664 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:23.108638    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20c97e3c-7ac2-49f5-9391-2ac5f64d8766" path="/var/lib/kubelet/pods/20c97e3c-7ac2-49f5-9391-2ac5f64d8766/volumes"
Nov 03 02:02:23.109079 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:23.109062    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33f9915e-45ca-4ab4-8466-0b4967abed94" path="/var/lib/kubelet/pods/33f9915e-45ca-4ab4-8466-0b4967abed94/volumes"
Nov 03 02:02:25.566619 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:25.566588    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"]
Nov 03 02:02:25.567069 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:25.566839    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv" podUID="f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc" containerName="maas-api" containerID="cri-o://342612a2affb16a164dd139e541d602329476ed595d07d0c451d9237a0e51542" gracePeriod=30
Nov 03 02:02:25.714281 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:25.714250    5005 generic.go:356] "Generic (PLEG): container finished" podID="f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc" containerID="342612a2affb16a164dd139e541d602329476ed595d07d0c451d9237a0e51542" exitCode=0
Nov 03 02:02:25.714411 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:25.714297    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv" event={"ID":"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc","Type":"ContainerDied","Data":"342612a2affb16a164dd139e541d602329476ed595d07d0c451d9237a0e51542"}
Nov 03 02:02:25.850267 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:25.850244    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"
Nov 03 02:02:25.953773 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:25.953745    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-metrics-certs\") pod \"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc\" (UID: \"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc\") "
Nov 03 02:02:25.953942 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:25.953782    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-maas-api-tls\") pod \"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc\" (UID: \"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc\") "
Nov 03 02:02:25.953942 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:25.953831    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-openshift-service-ca\") pod \"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc\" (UID: \"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc\") "
Nov 03 02:02:25.953942 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:25.953894    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-kube-api-access-wllgl\" (UniqueName: \"kubernetes.io/projected/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-kube-api-access-wllgl\") pod \"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc\" (UID: \"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc\") "
Nov 03 02:02:25.954177 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:25.954156    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-openshift-service-ca" pod "f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc" (UID: "f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:25.954310 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:25.954288    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:25.956106 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:25.956077    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-maas-api-tls" pod "f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc" (UID: "f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:25.956209 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:25.956129    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-metrics-certs" pod "f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc" (UID: "f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:25.956209 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:25.956145    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-kube-api-access-wllgl" pod "f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc" (UID: "f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc"). InnerVolumeSpecName "kube-api-access-wllgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:26.055232 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:26.055207    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:26.055232 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:26.055229    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:26.055408 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:26.055239    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wllgl\" (UniqueName: \"kubernetes.io/projected/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc-kube-api-access-wllgl\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:26.721105 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:26.721071    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv" event={"ID":"f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc","Type":"ContainerDied","Data":"52f911af679e82b51aeb0b9a85d2951f7b7ebf132106858b262d0bcd4c9a6dc3"}
Nov 03 02:02:26.721105 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:26.721102    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"
Nov 03 02:02:26.721593 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:26.721130    5005 scope.go:122] "RemoveContainer" containerID="342612a2affb16a164dd139e541d602329476ed595d07d0c451d9237a0e51542"
Nov 03 02:02:26.873620 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:26.873585    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"]
Nov 03 02:02:26.885808 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:26.885783    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fbf24ed8-5b6bbc99b8-dk2sv"]
Nov 03 02:02:27.111808 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:27.111780    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc" path="/var/lib/kubelet/pods/f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc/volumes"
Nov 03 02:02:30.069399 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:30.069364    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79"]
Nov 03 02:02:30.183651 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:30.183613    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9"]
Nov 03 02:02:30.712955 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:02:30.712918    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:02:32.353727 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.353687    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"]
Nov 03 02:02:32.354172 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.354158    5005 cpu_manager.go:414] "containerMap: removing container" podUID="89afc39a-86cd-46e9-bb50-bd3a3e2d0007" containerName="maas-api"
Nov 03 02:02:32.354218 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.354172    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89afc39a-86cd-46e9-bb50-bd3a3e2d0007" containerName="maas-api"
Nov 03 02:02:32.354218 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.354191    5005 cpu_manager.go:414] "containerMap: removing container" podUID="882bac36-975d-43b0-997d-e32a11b91009" containerName="maas-api"
Nov 03 02:02:32.354218 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.354196    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="882bac36-975d-43b0-997d-e32a11b91009" containerName="maas-api"
Nov 03 02:02:32.354218 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.354204    5005 cpu_manager.go:414] "containerMap: removing container" podUID="f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc" containerName="maas-api"
Nov 03 02:02:32.354218 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.354208    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc" containerName="maas-api"
Nov 03 02:02:32.354361 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.354226    5005 cpu_manager.go:414] "containerMap: removing container" podUID="0275219a-ec56-447b-9b83-19ec8008a85f" containerName="payload-pre-processing"
Nov 03 02:02:32.354361 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.354232    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0275219a-ec56-447b-9b83-19ec8008a85f" containerName="payload-pre-processing"
Nov 03 02:02:32.354361 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.354240    5005 cpu_manager.go:414] "containerMap: removing container" podUID="04b6c270-905c-467d-b07d-6d22ebf79a7b" containerName="payload-processing"
Nov 03 02:02:32.354361 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.354244    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b6c270-905c-467d-b07d-6d22ebf79a7b" containerName="payload-processing"
Nov 03 02:02:32.354361 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.354255    5005 cpu_manager.go:414] "containerMap: removing container" podUID="20c97e3c-7ac2-49f5-9391-2ac5f64d8766" containerName="payload-processing"
Nov 03 02:02:32.354361 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.354259    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c97e3c-7ac2-49f5-9391-2ac5f64d8766" containerName="payload-processing"
Nov 03 02:02:32.354361 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.354268    5005 cpu_manager.go:414] "containerMap: removing container" podUID="33f9915e-45ca-4ab4-8466-0b4967abed94" containerName="payload-pre-processing"
Nov 03 02:02:32.354361 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.354272    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f9915e-45ca-4ab4-8466-0b4967abed94" containerName="payload-pre-processing"
Nov 03 02:02:32.354585 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.354363    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="04b6c270-905c-467d-b07d-6d22ebf79a7b" containerName="payload-processing"
Nov 03 02:02:32.354585 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.354374    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c97e3c-7ac2-49f5-9391-2ac5f64d8766" containerName="payload-processing"
Nov 03 02:02:32.354585 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.354380    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f9915e-45ca-4ab4-8466-0b4967abed94" containerName="payload-pre-processing"
Nov 03 02:02:32.354585 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.354387    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="89afc39a-86cd-46e9-bb50-bd3a3e2d0007" containerName="maas-api"
Nov 03 02:02:32.354585 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.354395    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="882bac36-975d-43b0-997d-e32a11b91009" containerName="maas-api"
Nov 03 02:02:32.354585 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.354400    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8ed7bcd-76f1-45a6-8d34-a62e9eae70cc" containerName="maas-api"
Nov 03 02:02:32.354585 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.354407    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="0275219a-ec56-447b-9b83-19ec8008a85f" containerName="payload-pre-processing"
Nov 03 02:02:32.359113 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.359092    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"
Nov 03 02:02:32.410342 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.410321    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-521f1f-serving-cert\""
Nov 03 02:02:32.412175 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.412152    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"]
Nov 03 02:02:32.515203 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.515179    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw\" (UID: \"81bdbd55-f2ba-4334-8c9f-42d61fea3b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"
Nov 03 02:02:32.515348 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.515242    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnlxq\" (UniqueName: \"kubernetes.io/projected/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-kube-api-access-xnlxq\") pod \"maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw\" (UID: \"81bdbd55-f2ba-4334-8c9f-42d61fea3b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"
Nov 03 02:02:32.515348 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.515340    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-metrics-certs\") pod \"maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw\" (UID: \"81bdbd55-f2ba-4334-8c9f-42d61fea3b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"
Nov 03 02:02:32.515433 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.515383    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-maas-api-tls\") pod \"maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw\" (UID: \"81bdbd55-f2ba-4334-8c9f-42d61fea3b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"
Nov 03 02:02:32.616416 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.616350    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-metrics-certs\") pod \"maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw\" (UID: \"81bdbd55-f2ba-4334-8c9f-42d61fea3b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"
Nov 03 02:02:32.616416 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.616398    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-maas-api-tls\") pod \"maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw\" (UID: \"81bdbd55-f2ba-4334-8c9f-42d61fea3b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"
Nov 03 02:02:32.616623 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.616439    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw\" (UID: \"81bdbd55-f2ba-4334-8c9f-42d61fea3b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"
Nov 03 02:02:32.616623 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.616518    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnlxq\" (UniqueName: \"kubernetes.io/projected/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-kube-api-access-xnlxq\") pod \"maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw\" (UID: \"81bdbd55-f2ba-4334-8c9f-42d61fea3b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"
Nov 03 02:02:32.616623 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:02:32.616565    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-serving-cert: secret "maas-api-e2e-worker-w2-521f1f-serving-cert" not found
Nov 03 02:02:32.616799 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:02:32.616643    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-maas-api-tls podName:81bdbd55-f2ba-4334-8c9f-42d61fea3b33 nodeName:}" failed. No retries permitted until 2026-11-03 02:02:33.116617379 +0000 UTC m=+680.130858742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-maas-api-tls") pod "maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw" (UID: "81bdbd55-f2ba-4334-8c9f-42d61fea3b33") : secret "maas-api-e2e-worker-w2-521f1f-serving-cert" not found
Nov 03 02:02:32.617261 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.617238    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw\" (UID: \"81bdbd55-f2ba-4334-8c9f-42d61fea3b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"
Nov 03 02:02:32.618911 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.618885    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-metrics-certs\") pod \"maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw\" (UID: \"81bdbd55-f2ba-4334-8c9f-42d61fea3b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"
Nov 03 02:02:32.671889 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:32.671859    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnlxq\" (UniqueName: \"kubernetes.io/projected/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-kube-api-access-xnlxq\") pod \"maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw\" (UID: \"81bdbd55-f2ba-4334-8c9f-42d61fea3b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"
Nov 03 02:02:33.121442 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:33.121408    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-maas-api-tls\") pod \"maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw\" (UID: \"81bdbd55-f2ba-4334-8c9f-42d61fea3b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"
Nov 03 02:02:33.123860 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:33.123832    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-maas-api-tls\") pod \"maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw\" (UID: \"81bdbd55-f2ba-4334-8c9f-42d61fea3b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"
Nov 03 02:02:33.272134 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:33.272105    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"
Nov 03 02:02:33.644257 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:33.644224    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"]
Nov 03 02:02:33.650286 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:02:33.650254    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81bdbd55_f2ba_4334_8c9f_42d61fea3b33.slice/crio-36aa2c0ea1157f4bd01b9dd1218c86e250cd48b6f88e19985f0d7ab7262f8644 WatchSource:0}: Error finding container 36aa2c0ea1157f4bd01b9dd1218c86e250cd48b6f88e19985f0d7ab7262f8644: Status 404 returned error can't find the container with id 36aa2c0ea1157f4bd01b9dd1218c86e250cd48b6f88e19985f0d7ab7262f8644
Nov 03 02:02:33.762424 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:33.762393    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw" event={"ID":"81bdbd55-f2ba-4334-8c9f-42d61fea3b33","Type":"ContainerStarted","Data":"36aa2c0ea1157f4bd01b9dd1218c86e250cd48b6f88e19985f0d7ab7262f8644"}
Nov 03 02:02:34.770834 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:34.770778    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw" event={"ID":"81bdbd55-f2ba-4334-8c9f-42d61fea3b33","Type":"ContainerStarted","Data":"c33571d4623196ad2774f4015575df5ca45e881aac105368884d7bc40243b712"}
Nov 03 02:02:34.771281 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:34.770937    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"
Nov 03 02:02:34.879177 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:34.879091    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw" podStartSLOduration=1.9474849060000001 podStartE2EDuration="2.879079378s" totalImagesPullingTime="931.594472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:34.876337495 +0000 UTC m=+681.890578860" watchObservedRunningTime="2026-11-03 02:02:34.879079378 +0000 UTC m=+681.893320743"
Nov 03 02:02:35.070452 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.070409    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79" podUID="f10d4192-7fdc-4477-af07-2e37036a6595" containerName="payload-processing" containerID="cri-o://6ac11694535c6b11a7af18d2ad37351b2bdc77fbc12d3a29ccc356c08c6a4fe2" gracePeriod=25
Nov 03 02:02:35.185006 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.184914    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9" podUID="97f5dd4f-0053-4481-a27c-987237539a88" containerName="payload-pre-processing" containerID="cri-o://31175b71ef44e286be73f22e4abecd598e1595f6b5b2031f88fb47f871ee17cc" gracePeriod=25
Nov 03 02:02:35.341900 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.341869    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79"
Nov 03 02:02:35.439099 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.439048    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9"
Nov 03 02:02:35.443264 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.443234    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f10d4192-7fdc-4477-af07-2e37036a6595-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10d4192-7fdc-4477-af07-2e37036a6595-plugins-config-volume\") pod \"f10d4192-7fdc-4477-af07-2e37036a6595\" (UID: \"f10d4192-7fdc-4477-af07-2e37036a6595\") "
Nov 03 02:02:35.443361 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.443278    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10d4192-7fdc-4477-af07-2e37036a6595-kube-api-access-xn2nt\" (UniqueName: \"kubernetes.io/projected/f10d4192-7fdc-4477-af07-2e37036a6595-kube-api-access-xn2nt\") pod \"f10d4192-7fdc-4477-af07-2e37036a6595\" (UID: \"f10d4192-7fdc-4477-af07-2e37036a6595\") "
Nov 03 02:02:35.443638 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.443614    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f10d4192-7fdc-4477-af07-2e37036a6595-plugins-config-volume" pod "f10d4192-7fdc-4477-af07-2e37036a6595" (UID: "f10d4192-7fdc-4477-af07-2e37036a6595"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:35.445256 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.445235    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10d4192-7fdc-4477-af07-2e37036a6595-kube-api-access-xn2nt" pod "f10d4192-7fdc-4477-af07-2e37036a6595" (UID: "f10d4192-7fdc-4477-af07-2e37036a6595"). InnerVolumeSpecName "kube-api-access-xn2nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:35.544626 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.544600    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f5dd4f-0053-4481-a27c-987237539a88-kube-api-access-t9n57\" (UniqueName: \"kubernetes.io/projected/97f5dd4f-0053-4481-a27c-987237539a88-kube-api-access-t9n57\") pod \"97f5dd4f-0053-4481-a27c-987237539a88\" (UID: \"97f5dd4f-0053-4481-a27c-987237539a88\") "
Nov 03 02:02:35.544786 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.544760    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f5dd4f-0053-4481-a27c-987237539a88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f5dd4f-0053-4481-a27c-987237539a88-plugins-config-volume\") pod \"97f5dd4f-0053-4481-a27c-987237539a88\" (UID: \"97f5dd4f-0053-4481-a27c-987237539a88\") "
Nov 03 02:02:35.544964 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.544949    5005 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10d4192-7fdc-4477-af07-2e37036a6595-plugins-config-volume\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:35.545040 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.544963    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn2nt\" (UniqueName: \"kubernetes.io/projected/f10d4192-7fdc-4477-af07-2e37036a6595-kube-api-access-xn2nt\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:35.545102 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.545081    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f5dd4f-0053-4481-a27c-987237539a88-plugins-config-volume" pod "97f5dd4f-0053-4481-a27c-987237539a88" (UID: "97f5dd4f-0053-4481-a27c-987237539a88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:35.546686 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.546663    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f5dd4f-0053-4481-a27c-987237539a88-kube-api-access-t9n57" pod "97f5dd4f-0053-4481-a27c-987237539a88" (UID: "97f5dd4f-0053-4481-a27c-987237539a88"). InnerVolumeSpecName "kube-api-access-t9n57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:35.645593 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.645560    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9n57\" (UniqueName: \"kubernetes.io/projected/97f5dd4f-0053-4481-a27c-987237539a88-kube-api-access-t9n57\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:35.645593 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.645589    5005 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f5dd4f-0053-4481-a27c-987237539a88-plugins-config-volume\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:35.777550 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.777522    5005 generic.go:356] "Generic (PLEG): container finished" podID="f10d4192-7fdc-4477-af07-2e37036a6595" containerID="6ac11694535c6b11a7af18d2ad37351b2bdc77fbc12d3a29ccc356c08c6a4fe2" exitCode=0
Nov 03 02:02:35.778030 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.777579    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79"
Nov 03 02:02:35.778030 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.777602    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79" event={"ID":"f10d4192-7fdc-4477-af07-2e37036a6595","Type":"ContainerDied","Data":"6ac11694535c6b11a7af18d2ad37351b2bdc77fbc12d3a29ccc356c08c6a4fe2"}
Nov 03 02:02:35.778030 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.777682    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79" event={"ID":"f10d4192-7fdc-4477-af07-2e37036a6595","Type":"ContainerDied","Data":"07edb1b71ce599393af28a94580a32241933865e8ead31c5e50ff67e49c7f4df"}
Nov 03 02:02:35.778030 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.777733    5005 scope.go:122] "RemoveContainer" containerID="6ac11694535c6b11a7af18d2ad37351b2bdc77fbc12d3a29ccc356c08c6a4fe2"
Nov 03 02:02:35.779159 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.779136    5005 generic.go:356] "Generic (PLEG): container finished" podID="97f5dd4f-0053-4481-a27c-987237539a88" containerID="31175b71ef44e286be73f22e4abecd598e1595f6b5b2031f88fb47f871ee17cc" exitCode=0
Nov 03 02:02:35.779244 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.779191    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9"
Nov 03 02:02:35.779364 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.779217    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9" event={"ID":"97f5dd4f-0053-4481-a27c-987237539a88","Type":"ContainerDied","Data":"31175b71ef44e286be73f22e4abecd598e1595f6b5b2031f88fb47f871ee17cc"}
Nov 03 02:02:35.779441 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.779395    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9" event={"ID":"97f5dd4f-0053-4481-a27c-987237539a88","Type":"ContainerDied","Data":"bd6f8a4e04bf1195495305619c1b39990eb27d0f55d79173e3453c103a07ef4d"}
Nov 03 02:02:35.789036 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.789021    5005 scope.go:122] "RemoveContainer" containerID="6ac11694535c6b11a7af18d2ad37351b2bdc77fbc12d3a29ccc356c08c6a4fe2"
Nov 03 02:02:35.789315 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:02:35.789297    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ac11694535c6b11a7af18d2ad37351b2bdc77fbc12d3a29ccc356c08c6a4fe2\": container with ID starting with 6ac11694535c6b11a7af18d2ad37351b2bdc77fbc12d3a29ccc356c08c6a4fe2 not found: ID does not exist" containerID="6ac11694535c6b11a7af18d2ad37351b2bdc77fbc12d3a29ccc356c08c6a4fe2"
Nov 03 02:02:35.789402 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.789325    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ac11694535c6b11a7af18d2ad37351b2bdc77fbc12d3a29ccc356c08c6a4fe2"} err="failed to get container status \"6ac11694535c6b11a7af18d2ad37351b2bdc77fbc12d3a29ccc356c08c6a4fe2\": rpc error: code = NotFound desc = could not find container \"6ac11694535c6b11a7af18d2ad37351b2bdc77fbc12d3a29ccc356c08c6a4fe2\": container with ID starting with 6ac11694535c6b11a7af18d2ad37351b2bdc77fbc12d3a29ccc356c08c6a4fe2 not found: ID does not exist"
Nov 03 02:02:35.789402 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.789349    5005 scope.go:122] "RemoveContainer" containerID="31175b71ef44e286be73f22e4abecd598e1595f6b5b2031f88fb47f871ee17cc"
Nov 03 02:02:35.798009 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.797991    5005 scope.go:122] "RemoveContainer" containerID="31175b71ef44e286be73f22e4abecd598e1595f6b5b2031f88fb47f871ee17cc"
Nov 03 02:02:35.798266 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:02:35.798247    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31175b71ef44e286be73f22e4abecd598e1595f6b5b2031f88fb47f871ee17cc\": container with ID starting with 31175b71ef44e286be73f22e4abecd598e1595f6b5b2031f88fb47f871ee17cc not found: ID does not exist" containerID="31175b71ef44e286be73f22e4abecd598e1595f6b5b2031f88fb47f871ee17cc"
Nov 03 02:02:35.798313 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.798272    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31175b71ef44e286be73f22e4abecd598e1595f6b5b2031f88fb47f871ee17cc"} err="failed to get container status \"31175b71ef44e286be73f22e4abecd598e1595f6b5b2031f88fb47f871ee17cc\": rpc error: code = NotFound desc = could not find container \"31175b71ef44e286be73f22e4abecd598e1595f6b5b2031f88fb47f871ee17cc\": container with ID starting with 31175b71ef44e286be73f22e4abecd598e1595f6b5b2031f88fb47f871ee17cc not found: ID does not exist"
Nov 03 02:02:35.930467 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.930437    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79"]
Nov 03 02:02:35.935280 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:35.935253    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-080015ba-5cbbd54cf7-tcl79"]
Nov 03 02:02:36.035617 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:36.035554    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9"]
Nov 03 02:02:36.038559 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:36.038536    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-080015ba-76bcb64fbc-kwbn9"]
Nov 03 02:02:37.108777 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.108735    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f5dd4f-0053-4481-a27c-987237539a88" path="/var/lib/kubelet/pods/97f5dd4f-0053-4481-a27c-987237539a88/volumes"
Nov 03 02:02:37.109188 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.109167    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f10d4192-7fdc-4477-af07-2e37036a6595" path="/var/lib/kubelet/pods/f10d4192-7fdc-4477-af07-2e37036a6595/volumes"
Nov 03 02:02:37.191067 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.191040    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"]
Nov 03 02:02:37.191262 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.191218    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7" podUID="6234a2b0-8d37-4048-ab0b-27029c078d1d" containerName="maas-api" containerID="cri-o://076f0b7756f44c023c4f36771ed593f67f83861bcc927e1aff328d9f0458192a" gracePeriod=30
Nov 03 02:02:37.355508 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.355469    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7" podUID="6234a2b0-8d37-4048-ab0b-27029c078d1d" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.67:8443/health\": dial tcp 10.132.0.67:8443: connect: connection refused"
Nov 03 02:02:37.444212 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.444190    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"
Nov 03 02:02:37.563052 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.563025    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6234a2b0-8d37-4048-ab0b-27029c078d1d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6234a2b0-8d37-4048-ab0b-27029c078d1d-metrics-certs\") pod \"6234a2b0-8d37-4048-ab0b-27029c078d1d\" (UID: \"6234a2b0-8d37-4048-ab0b-27029c078d1d\") "
Nov 03 02:02:37.563177 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.563090    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6234a2b0-8d37-4048-ab0b-27029c078d1d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6234a2b0-8d37-4048-ab0b-27029c078d1d-maas-api-tls\") pod \"6234a2b0-8d37-4048-ab0b-27029c078d1d\" (UID: \"6234a2b0-8d37-4048-ab0b-27029c078d1d\") "
Nov 03 02:02:37.563251 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.563195    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6234a2b0-8d37-4048-ab0b-27029c078d1d-kube-api-access-b5q8n\" (UniqueName: \"kubernetes.io/projected/6234a2b0-8d37-4048-ab0b-27029c078d1d-kube-api-access-b5q8n\") pod \"6234a2b0-8d37-4048-ab0b-27029c078d1d\" (UID: \"6234a2b0-8d37-4048-ab0b-27029c078d1d\") "
Nov 03 02:02:37.563307 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.563254    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6234a2b0-8d37-4048-ab0b-27029c078d1d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6234a2b0-8d37-4048-ab0b-27029c078d1d-openshift-service-ca\") pod \"6234a2b0-8d37-4048-ab0b-27029c078d1d\" (UID: \"6234a2b0-8d37-4048-ab0b-27029c078d1d\") "
Nov 03 02:02:37.563730 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.563682    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6234a2b0-8d37-4048-ab0b-27029c078d1d-openshift-service-ca" pod "6234a2b0-8d37-4048-ab0b-27029c078d1d" (UID: "6234a2b0-8d37-4048-ab0b-27029c078d1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:37.565495 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.565468    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6234a2b0-8d37-4048-ab0b-27029c078d1d-kube-api-access-b5q8n" pod "6234a2b0-8d37-4048-ab0b-27029c078d1d" (UID: "6234a2b0-8d37-4048-ab0b-27029c078d1d"). InnerVolumeSpecName "kube-api-access-b5q8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:37.566142 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.566105    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6234a2b0-8d37-4048-ab0b-27029c078d1d-metrics-certs" pod "6234a2b0-8d37-4048-ab0b-27029c078d1d" (UID: "6234a2b0-8d37-4048-ab0b-27029c078d1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:37.566369 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.566345    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6234a2b0-8d37-4048-ab0b-27029c078d1d-maas-api-tls" pod "6234a2b0-8d37-4048-ab0b-27029c078d1d" (UID: "6234a2b0-8d37-4048-ab0b-27029c078d1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:37.664594 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.664566    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6234a2b0-8d37-4048-ab0b-27029c078d1d-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:37.664594 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.664593    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6234a2b0-8d37-4048-ab0b-27029c078d1d-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:37.664745 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.664604    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5q8n\" (UniqueName: \"kubernetes.io/projected/6234a2b0-8d37-4048-ab0b-27029c078d1d-kube-api-access-b5q8n\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:37.664745 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.664617    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6234a2b0-8d37-4048-ab0b-27029c078d1d-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:37.794210 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.794178    5005 generic.go:356] "Generic (PLEG): container finished" podID="6234a2b0-8d37-4048-ab0b-27029c078d1d" containerID="076f0b7756f44c023c4f36771ed593f67f83861bcc927e1aff328d9f0458192a" exitCode=0
Nov 03 02:02:37.794356 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.794230    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"
Nov 03 02:02:37.794356 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.794247    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7" event={"ID":"6234a2b0-8d37-4048-ab0b-27029c078d1d","Type":"ContainerDied","Data":"076f0b7756f44c023c4f36771ed593f67f83861bcc927e1aff328d9f0458192a"}
Nov 03 02:02:37.794356 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.794293    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7" event={"ID":"6234a2b0-8d37-4048-ab0b-27029c078d1d","Type":"ContainerDied","Data":"58402a9c9568c9bf7c22b861cf2a5dce28b2940430ff385168eb58f8ffd60511"}
Nov 03 02:02:37.794356 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.794313    5005 scope.go:122] "RemoveContainer" containerID="076f0b7756f44c023c4f36771ed593f67f83861bcc927e1aff328d9f0458192a"
Nov 03 02:02:37.804711 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.804679    5005 scope.go:122] "RemoveContainer" containerID="076f0b7756f44c023c4f36771ed593f67f83861bcc927e1aff328d9f0458192a"
Nov 03 02:02:37.805001 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:02:37.804979    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"076f0b7756f44c023c4f36771ed593f67f83861bcc927e1aff328d9f0458192a\": container with ID starting with 076f0b7756f44c023c4f36771ed593f67f83861bcc927e1aff328d9f0458192a not found: ID does not exist" containerID="076f0b7756f44c023c4f36771ed593f67f83861bcc927e1aff328d9f0458192a"
Nov 03 02:02:37.805048 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.805014    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"076f0b7756f44c023c4f36771ed593f67f83861bcc927e1aff328d9f0458192a"} err="failed to get container status \"076f0b7756f44c023c4f36771ed593f67f83861bcc927e1aff328d9f0458192a\": rpc error: code = NotFound desc = could not find container \"076f0b7756f44c023c4f36771ed593f67f83861bcc927e1aff328d9f0458192a\": container with ID starting with 076f0b7756f44c023c4f36771ed593f67f83861bcc927e1aff328d9f0458192a not found: ID does not exist"
Nov 03 02:02:37.947104 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.947079    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"]
Nov 03 02:02:37.949838 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:37.949816    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aead6b17-596bf9ff59-b49g7"]
Nov 03 02:02:38.854950 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:38.854920    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"]
Nov 03 02:02:38.855404 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:38.855390    5005 cpu_manager.go:414] "containerMap: removing container" podUID="6234a2b0-8d37-4048-ab0b-27029c078d1d" containerName="maas-api"
Nov 03 02:02:38.855457 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:38.855406    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6234a2b0-8d37-4048-ab0b-27029c078d1d" containerName="maas-api"
Nov 03 02:02:38.855457 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:38.855420    5005 cpu_manager.go:414] "containerMap: removing container" podUID="97f5dd4f-0053-4481-a27c-987237539a88" containerName="payload-pre-processing"
Nov 03 02:02:38.855457 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:38.855426    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f5dd4f-0053-4481-a27c-987237539a88" containerName="payload-pre-processing"
Nov 03 02:02:38.855457 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:38.855441    5005 cpu_manager.go:414] "containerMap: removing container" podUID="f10d4192-7fdc-4477-af07-2e37036a6595" containerName="payload-processing"
Nov 03 02:02:38.855457 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:38.855446    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10d4192-7fdc-4477-af07-2e37036a6595" containerName="payload-processing"
Nov 03 02:02:38.855617 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:38.855549    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10d4192-7fdc-4477-af07-2e37036a6595" containerName="payload-processing"
Nov 03 02:02:38.855617 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:38.855561    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="6234a2b0-8d37-4048-ab0b-27029c078d1d" containerName="maas-api"
Nov 03 02:02:38.855617 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:38.855569    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f5dd4f-0053-4481-a27c-987237539a88" containerName="payload-pre-processing"
Nov 03 02:02:38.860156 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:38.860134    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"
Nov 03 02:02:38.910493 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:38.910473    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-a0e72e-serving-cert\""
Nov 03 02:02:38.913909 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:38.913888    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"]
Nov 03 02:02:38.978052 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:38.978027    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43baeed2-8514-4f19-90c6-e74f971f32b0-metrics-certs\") pod \"maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k\" (UID: \"43baeed2-8514-4f19-90c6-e74f971f32b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"
Nov 03 02:02:38.978138 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:38.978057    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43baeed2-8514-4f19-90c6-e74f971f32b0-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k\" (UID: \"43baeed2-8514-4f19-90c6-e74f971f32b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"
Nov 03 02:02:38.978174 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:38.978161    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wkwt\" (UniqueName: \"kubernetes.io/projected/43baeed2-8514-4f19-90c6-e74f971f32b0-kube-api-access-4wkwt\") pod \"maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k\" (UID: \"43baeed2-8514-4f19-90c6-e74f971f32b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"
Nov 03 02:02:38.978218 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:38.978196    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43baeed2-8514-4f19-90c6-e74f971f32b0-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k\" (UID: \"43baeed2-8514-4f19-90c6-e74f971f32b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"
Nov 03 02:02:39.079602 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:39.079574    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wkwt\" (UniqueName: \"kubernetes.io/projected/43baeed2-8514-4f19-90c6-e74f971f32b0-kube-api-access-4wkwt\") pod \"maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k\" (UID: \"43baeed2-8514-4f19-90c6-e74f971f32b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"
Nov 03 02:02:39.079736 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:39.079612    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43baeed2-8514-4f19-90c6-e74f971f32b0-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k\" (UID: \"43baeed2-8514-4f19-90c6-e74f971f32b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"
Nov 03 02:02:39.079736 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:39.079681    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43baeed2-8514-4f19-90c6-e74f971f32b0-metrics-certs\") pod \"maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k\" (UID: \"43baeed2-8514-4f19-90c6-e74f971f32b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"
Nov 03 02:02:39.079736 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:39.079722    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43baeed2-8514-4f19-90c6-e74f971f32b0-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k\" (UID: \"43baeed2-8514-4f19-90c6-e74f971f32b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"
Nov 03 02:02:39.079933 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:02:39.079902    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-serving-cert: secret "maas-api-e2e-worker-w3-a0e72e-serving-cert" not found
Nov 03 02:02:39.080039 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:02:39.079986    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43baeed2-8514-4f19-90c6-e74f971f32b0-maas-api-tls podName:43baeed2-8514-4f19-90c6-e74f971f32b0 nodeName:}" failed. No retries permitted until 2026-11-03 02:02:39.579966269 +0000 UTC m=+686.594207629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/43baeed2-8514-4f19-90c6-e74f971f32b0-maas-api-tls") pod "maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k" (UID: "43baeed2-8514-4f19-90c6-e74f971f32b0") : secret "maas-api-e2e-worker-w3-a0e72e-serving-cert" not found
Nov 03 02:02:39.080385 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:39.080369    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43baeed2-8514-4f19-90c6-e74f971f32b0-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k\" (UID: \"43baeed2-8514-4f19-90c6-e74f971f32b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"
Nov 03 02:02:39.082079 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:39.082060    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43baeed2-8514-4f19-90c6-e74f971f32b0-metrics-certs\") pod \"maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k\" (UID: \"43baeed2-8514-4f19-90c6-e74f971f32b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"
Nov 03 02:02:39.115240 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:39.115164    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6234a2b0-8d37-4048-ab0b-27029c078d1d" path="/var/lib/kubelet/pods/6234a2b0-8d37-4048-ab0b-27029c078d1d/volumes"
Nov 03 02:02:39.135903 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:39.135786    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wkwt\" (UniqueName: \"kubernetes.io/projected/43baeed2-8514-4f19-90c6-e74f971f32b0-kube-api-access-4wkwt\") pod \"maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k\" (UID: \"43baeed2-8514-4f19-90c6-e74f971f32b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"
Nov 03 02:02:39.585556 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:39.585519    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43baeed2-8514-4f19-90c6-e74f971f32b0-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k\" (UID: \"43baeed2-8514-4f19-90c6-e74f971f32b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"
Nov 03 02:02:39.587799 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:39.587770    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43baeed2-8514-4f19-90c6-e74f971f32b0-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k\" (UID: \"43baeed2-8514-4f19-90c6-e74f971f32b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"
Nov 03 02:02:39.773804 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:39.773781    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"
Nov 03 02:02:40.146533 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:40.146505    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"]
Nov 03 02:02:40.152497 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:02:40.152473    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43baeed2_8514_4f19_90c6_e74f971f32b0.slice/crio-c57c291f4619bba92109f53ac9f9c570e17c222f46929082ed830c3c602dd620 WatchSource:0}: Error finding container c57c291f4619bba92109f53ac9f9c570e17c222f46929082ed830c3c602dd620: Status 404 returned error can't find the container with id c57c291f4619bba92109f53ac9f9c570e17c222f46929082ed830c3c602dd620
Nov 03 02:02:40.784835 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:40.784800    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"
Nov 03 02:02:40.815211 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:40.815172    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k" event={"ID":"43baeed2-8514-4f19-90c6-e74f971f32b0","Type":"ContainerStarted","Data":"c57c291f4619bba92109f53ac9f9c570e17c222f46929082ed830c3c602dd620"}
Nov 03 02:02:41.822762 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:41.822726    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k" event={"ID":"43baeed2-8514-4f19-90c6-e74f971f32b0","Type":"ContainerStarted","Data":"037c1cbf55366f1451cb52d57fc3c6838876bd1927247a1e1487d0c9f7ab845e"}
Nov 03 02:02:41.823155 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:41.822888    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"
Nov 03 02:02:41.952748 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:41.952678    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k" podStartSLOduration=3.051530392 podStartE2EDuration="3.95266437s" totalImagesPullingTime="901.133978ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:41.95094555 +0000 UTC m=+688.965186916" watchObservedRunningTime="2026-11-03 02:02:41.95266437 +0000 UTC m=+688.966905735"
Nov 03 02:02:47.833990 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:47.833961    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"
Nov 03 02:02:48.612246 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:48.612210    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"]
Nov 03 02:02:48.612434 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:48.612413    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l" podUID="0fb8539d-0d8e-41a6-867d-ae58573b5e76" containerName="maas-api" containerID="cri-o://caeeea65576e6a9c1344ae4bb34a74da601c2d2469cf8bd35e92432f076eaeb7" gracePeriod=30
Nov 03 02:02:48.857635 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:48.857612    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"
Nov 03 02:02:48.868566 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:48.868516    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fb8539d-0d8e-41a6-867d-ae58573b5e76-kube-api-access-h6sbd\" (UniqueName: \"kubernetes.io/projected/0fb8539d-0d8e-41a6-867d-ae58573b5e76-kube-api-access-h6sbd\") pod \"0fb8539d-0d8e-41a6-867d-ae58573b5e76\" (UID: \"0fb8539d-0d8e-41a6-867d-ae58573b5e76\") "
Nov 03 02:02:48.868566 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:48.868563    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fb8539d-0d8e-41a6-867d-ae58573b5e76-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fb8539d-0d8e-41a6-867d-ae58573b5e76-maas-api-tls\") pod \"0fb8539d-0d8e-41a6-867d-ae58573b5e76\" (UID: \"0fb8539d-0d8e-41a6-867d-ae58573b5e76\") "
Nov 03 02:02:48.868794 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:48.868604    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fb8539d-0d8e-41a6-867d-ae58573b5e76-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb8539d-0d8e-41a6-867d-ae58573b5e76-openshift-service-ca\") pod \"0fb8539d-0d8e-41a6-867d-ae58573b5e76\" (UID: \"0fb8539d-0d8e-41a6-867d-ae58573b5e76\") "
Nov 03 02:02:48.868794 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:48.868766    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fb8539d-0d8e-41a6-867d-ae58573b5e76-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fb8539d-0d8e-41a6-867d-ae58573b5e76-metrics-certs\") pod \"0fb8539d-0d8e-41a6-867d-ae58573b5e76\" (UID: \"0fb8539d-0d8e-41a6-867d-ae58573b5e76\") "
Nov 03 02:02:48.869126 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:48.869097    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fb8539d-0d8e-41a6-867d-ae58573b5e76-openshift-service-ca" pod "0fb8539d-0d8e-41a6-867d-ae58573b5e76" (UID: "0fb8539d-0d8e-41a6-867d-ae58573b5e76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:48.869256 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:48.869190    5005 generic.go:356] "Generic (PLEG): container finished" podID="0fb8539d-0d8e-41a6-867d-ae58573b5e76" containerID="caeeea65576e6a9c1344ae4bb34a74da601c2d2469cf8bd35e92432f076eaeb7" exitCode=0
Nov 03 02:02:48.869256 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:48.869248    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"
Nov 03 02:02:48.869342 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:48.869281    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l" event={"ID":"0fb8539d-0d8e-41a6-867d-ae58573b5e76","Type":"ContainerDied","Data":"caeeea65576e6a9c1344ae4bb34a74da601c2d2469cf8bd35e92432f076eaeb7"}
Nov 03 02:02:48.869342 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:48.869331    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l" event={"ID":"0fb8539d-0d8e-41a6-867d-ae58573b5e76","Type":"ContainerDied","Data":"eb7eb58098da82fc6b43e1aaf699767d29e5c1e75a4fbebc8dff0c2b7510adbc"}
Nov 03 02:02:48.869403 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:48.869357    5005 scope.go:122] "RemoveContainer" containerID="caeeea65576e6a9c1344ae4bb34a74da601c2d2469cf8bd35e92432f076eaeb7"
Nov 03 02:02:48.870874 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:48.870844    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb8539d-0d8e-41a6-867d-ae58573b5e76-kube-api-access-h6sbd" pod "0fb8539d-0d8e-41a6-867d-ae58573b5e76" (UID: "0fb8539d-0d8e-41a6-867d-ae58573b5e76"). InnerVolumeSpecName "kube-api-access-h6sbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:48.871141 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:48.871125    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fb8539d-0d8e-41a6-867d-ae58573b5e76-maas-api-tls" pod "0fb8539d-0d8e-41a6-867d-ae58573b5e76" (UID: "0fb8539d-0d8e-41a6-867d-ae58573b5e76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:48.871212 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:48.871153    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fb8539d-0d8e-41a6-867d-ae58573b5e76-metrics-certs" pod "0fb8539d-0d8e-41a6-867d-ae58573b5e76" (UID: "0fb8539d-0d8e-41a6-867d-ae58573b5e76"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:48.893085 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:48.893068    5005 scope.go:122] "RemoveContainer" containerID="caeeea65576e6a9c1344ae4bb34a74da601c2d2469cf8bd35e92432f076eaeb7"
Nov 03 02:02:48.893377 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:02:48.893358    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caeeea65576e6a9c1344ae4bb34a74da601c2d2469cf8bd35e92432f076eaeb7\": container with ID starting with caeeea65576e6a9c1344ae4bb34a74da601c2d2469cf8bd35e92432f076eaeb7 not found: ID does not exist" containerID="caeeea65576e6a9c1344ae4bb34a74da601c2d2469cf8bd35e92432f076eaeb7"
Nov 03 02:02:48.893430 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:48.893386    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caeeea65576e6a9c1344ae4bb34a74da601c2d2469cf8bd35e92432f076eaeb7"} err="failed to get container status \"caeeea65576e6a9c1344ae4bb34a74da601c2d2469cf8bd35e92432f076eaeb7\": rpc error: code = NotFound desc = could not find container \"caeeea65576e6a9c1344ae4bb34a74da601c2d2469cf8bd35e92432f076eaeb7\": container with ID starting with caeeea65576e6a9c1344ae4bb34a74da601c2d2469cf8bd35e92432f076eaeb7 not found: ID does not exist"
Nov 03 02:02:48.970018 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:48.969992    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fb8539d-0d8e-41a6-867d-ae58573b5e76-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:48.970018 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:48.970018    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6sbd\" (UniqueName: \"kubernetes.io/projected/0fb8539d-0d8e-41a6-867d-ae58573b5e76-kube-api-access-h6sbd\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:48.970199 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:48.970029    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fb8539d-0d8e-41a6-867d-ae58573b5e76-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:48.970199 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:48.970040    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb8539d-0d8e-41a6-867d-ae58573b5e76-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:49.276095 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:49.276068    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"]
Nov 03 02:02:49.278220 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:49.278196    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-080015ba-7665dd8687-8k69l"]
Nov 03 02:02:50.395078 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:50.395051    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"]
Nov 03 02:02:50.395503 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:50.395490    5005 cpu_manager.go:414] "containerMap: removing container" podUID="0fb8539d-0d8e-41a6-867d-ae58573b5e76" containerName="maas-api"
Nov 03 02:02:50.395551 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:50.395504    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fb8539d-0d8e-41a6-867d-ae58573b5e76" containerName="maas-api"
Nov 03 02:02:50.395610 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:50.395601    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fb8539d-0d8e-41a6-867d-ae58573b5e76" containerName="maas-api"
Nov 03 02:02:50.400069 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:50.400047    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"
Nov 03 02:02:50.449874 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:50.449848    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-1bccbd-serving-cert\""
Nov 03 02:02:50.452264 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:50.452238    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"]
Nov 03 02:02:50.484443 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:50.484418    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/532d5d68-bc86-491b-978b-97f9cf54580a-maas-api-tls\") pod \"maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j\" (UID: \"532d5d68-bc86-491b-978b-97f9cf54580a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"
Nov 03 02:02:50.484557 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:50.484496    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/532d5d68-bc86-491b-978b-97f9cf54580a-metrics-certs\") pod \"maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j\" (UID: \"532d5d68-bc86-491b-978b-97f9cf54580a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"
Nov 03 02:02:50.484557 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:50.484549    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np68m\" (UniqueName: \"kubernetes.io/projected/532d5d68-bc86-491b-978b-97f9cf54580a-kube-api-access-np68m\") pod \"maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j\" (UID: \"532d5d68-bc86-491b-978b-97f9cf54580a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"
Nov 03 02:02:50.484651 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:50.484631    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/532d5d68-bc86-491b-978b-97f9cf54580a-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j\" (UID: \"532d5d68-bc86-491b-978b-97f9cf54580a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"
Nov 03 02:02:50.585748 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:50.585713    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np68m\" (UniqueName: \"kubernetes.io/projected/532d5d68-bc86-491b-978b-97f9cf54580a-kube-api-access-np68m\") pod \"maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j\" (UID: \"532d5d68-bc86-491b-978b-97f9cf54580a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"
Nov 03 02:02:50.585882 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:50.585782    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/532d5d68-bc86-491b-978b-97f9cf54580a-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j\" (UID: \"532d5d68-bc86-491b-978b-97f9cf54580a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"
Nov 03 02:02:50.585929 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:50.585892    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/532d5d68-bc86-491b-978b-97f9cf54580a-maas-api-tls\") pod \"maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j\" (UID: \"532d5d68-bc86-491b-978b-97f9cf54580a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"
Nov 03 02:02:50.585986 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:50.585960    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/532d5d68-bc86-491b-978b-97f9cf54580a-metrics-certs\") pod \"maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j\" (UID: \"532d5d68-bc86-491b-978b-97f9cf54580a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"
Nov 03 02:02:50.586036 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:02:50.586017    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-serving-cert: secret "maas-api-e2e-worker-w1-1bccbd-serving-cert" not found
Nov 03 02:02:50.586084 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:02:50.586076    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/532d5d68-bc86-491b-978b-97f9cf54580a-maas-api-tls podName:532d5d68-bc86-491b-978b-97f9cf54580a nodeName:}" failed. No retries permitted until 2026-11-03 02:02:51.086061937 +0000 UTC m=+698.100303285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/532d5d68-bc86-491b-978b-97f9cf54580a-maas-api-tls") pod "maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j" (UID: "532d5d68-bc86-491b-978b-97f9cf54580a") : secret "maas-api-e2e-worker-w1-1bccbd-serving-cert" not found
Nov 03 02:02:50.586401 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:50.586376    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/532d5d68-bc86-491b-978b-97f9cf54580a-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j\" (UID: \"532d5d68-bc86-491b-978b-97f9cf54580a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"
Nov 03 02:02:50.588498 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:50.588472    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/532d5d68-bc86-491b-978b-97f9cf54580a-metrics-certs\") pod \"maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j\" (UID: \"532d5d68-bc86-491b-978b-97f9cf54580a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"
Nov 03 02:02:50.640150 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:50.640124    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np68m\" (UniqueName: \"kubernetes.io/projected/532d5d68-bc86-491b-978b-97f9cf54580a-kube-api-access-np68m\") pod \"maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j\" (UID: \"532d5d68-bc86-491b-978b-97f9cf54580a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"
Nov 03 02:02:51.090853 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:51.090814    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/532d5d68-bc86-491b-978b-97f9cf54580a-maas-api-tls\") pod \"maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j\" (UID: \"532d5d68-bc86-491b-978b-97f9cf54580a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"
Nov 03 02:02:51.093176 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:51.093144    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/532d5d68-bc86-491b-978b-97f9cf54580a-maas-api-tls\") pod \"maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j\" (UID: \"532d5d68-bc86-491b-978b-97f9cf54580a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"
Nov 03 02:02:51.110118 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:51.110090    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fb8539d-0d8e-41a6-867d-ae58573b5e76" path="/var/lib/kubelet/pods/0fb8539d-0d8e-41a6-867d-ae58573b5e76/volumes"
Nov 03 02:02:51.312408 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:51.312378    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"
Nov 03 02:02:51.681954 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:51.681882    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"]
Nov 03 02:02:51.688167 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:02:51.688139    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532d5d68_bc86_491b_978b_97f9cf54580a.slice/crio-2d97df12d7689a56380cf493fa2681dd4a204e9ac265e104fc215f6de80293aa WatchSource:0}: Error finding container 2d97df12d7689a56380cf493fa2681dd4a204e9ac265e104fc215f6de80293aa: Status 404 returned error can't find the container with id 2d97df12d7689a56380cf493fa2681dd4a204e9ac265e104fc215f6de80293aa
Nov 03 02:02:51.888219 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:51.888185    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j" event={"ID":"532d5d68-bc86-491b-978b-97f9cf54580a","Type":"ContainerStarted","Data":"2d97df12d7689a56380cf493fa2681dd4a204e9ac265e104fc215f6de80293aa"}
Nov 03 02:02:52.895175 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:52.895100    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j" event={"ID":"532d5d68-bc86-491b-978b-97f9cf54580a","Type":"ContainerStarted","Data":"dbb9867a9bed6184188617785575b72a86e40235e37d56ead9ebb319bb6a6968"}
Nov 03 02:02:52.895554 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:52.895222    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"
Nov 03 02:02:53.003262 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:53.003207    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j" podStartSLOduration=2.069463177 podStartE2EDuration="3.003192765s" totalImagesPullingTime="933.729588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:53.002784784 +0000 UTC m=+700.017026151" watchObservedRunningTime="2026-11-03 02:02:53.003192765 +0000 UTC m=+700.017434130"
Nov 03 02:02:58.904634 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:02:58.904605    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"
Nov 03 02:03:00.713557 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:03:00.713515    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:03:10.961383 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:10.961352    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"]
Nov 03 02:03:10.965337 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:10.965318    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"
Nov 03 02:03:11.062854 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:11.062812    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-c7d758-serving-cert\""
Nov 03 02:03:11.065519 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:11.065499    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"]
Nov 03 02:03:11.077425 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:11.077404    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbdeec42-7115-4361-ac6f-1874ffded23c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w\" (UID: \"fbdeec42-7115-4361-ac6f-1874ffded23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"
Nov 03 02:03:11.077499 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:11.077433    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbdeec42-7115-4361-ac6f-1874ffded23c-metrics-certs\") pod \"maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w\" (UID: \"fbdeec42-7115-4361-ac6f-1874ffded23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"
Nov 03 02:03:11.077499 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:11.077452    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdeec42-7115-4361-ac6f-1874ffded23c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w\" (UID: \"fbdeec42-7115-4361-ac6f-1874ffded23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"
Nov 03 02:03:11.077586 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:11.077546    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9lqz\" (UniqueName: \"kubernetes.io/projected/fbdeec42-7115-4361-ac6f-1874ffded23c-kube-api-access-l9lqz\") pod \"maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w\" (UID: \"fbdeec42-7115-4361-ac6f-1874ffded23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"
Nov 03 02:03:11.178846 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:11.178814    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9lqz\" (UniqueName: \"kubernetes.io/projected/fbdeec42-7115-4361-ac6f-1874ffded23c-kube-api-access-l9lqz\") pod \"maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w\" (UID: \"fbdeec42-7115-4361-ac6f-1874ffded23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"
Nov 03 02:03:11.179012 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:11.178945    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbdeec42-7115-4361-ac6f-1874ffded23c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w\" (UID: \"fbdeec42-7115-4361-ac6f-1874ffded23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"
Nov 03 02:03:11.179012 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:11.178979    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbdeec42-7115-4361-ac6f-1874ffded23c-metrics-certs\") pod \"maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w\" (UID: \"fbdeec42-7115-4361-ac6f-1874ffded23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"
Nov 03 02:03:11.179012 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:11.179003    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdeec42-7115-4361-ac6f-1874ffded23c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w\" (UID: \"fbdeec42-7115-4361-ac6f-1874ffded23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"
Nov 03 02:03:11.179171 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:03:11.179077    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-serving-cert: secret "maas-api-e2e-worker-w4-c7d758-serving-cert" not found
Nov 03 02:03:11.179171 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:03:11.179141    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbdeec42-7115-4361-ac6f-1874ffded23c-maas-api-tls podName:fbdeec42-7115-4361-ac6f-1874ffded23c nodeName:}" failed. No retries permitted until 2026-11-03 02:03:11.679127077 +0000 UTC m=+718.693368421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fbdeec42-7115-4361-ac6f-1874ffded23c-maas-api-tls") pod "maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w" (UID: "fbdeec42-7115-4361-ac6f-1874ffded23c") : secret "maas-api-e2e-worker-w4-c7d758-serving-cert" not found
Nov 03 02:03:11.179629 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:11.179611    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdeec42-7115-4361-ac6f-1874ffded23c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w\" (UID: \"fbdeec42-7115-4361-ac6f-1874ffded23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"
Nov 03 02:03:11.181205 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:11.181189    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbdeec42-7115-4361-ac6f-1874ffded23c-metrics-certs\") pod \"maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w\" (UID: \"fbdeec42-7115-4361-ac6f-1874ffded23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"
Nov 03 02:03:11.232278 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:11.232205    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9lqz\" (UniqueName: \"kubernetes.io/projected/fbdeec42-7115-4361-ac6f-1874ffded23c-kube-api-access-l9lqz\") pod \"maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w\" (UID: \"fbdeec42-7115-4361-ac6f-1874ffded23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"
Nov 03 02:03:11.684560 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:11.684517    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbdeec42-7115-4361-ac6f-1874ffded23c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w\" (UID: \"fbdeec42-7115-4361-ac6f-1874ffded23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"
Nov 03 02:03:11.686893 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:11.686852    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbdeec42-7115-4361-ac6f-1874ffded23c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w\" (UID: \"fbdeec42-7115-4361-ac6f-1874ffded23c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"
Nov 03 02:03:11.878198 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:11.878164    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"
Nov 03 02:03:12.246870 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:12.246823    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"]
Nov 03 02:03:12.252880 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:03:12.252851    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbdeec42_7115_4361_ac6f_1874ffded23c.slice/crio-72835cd833dc59a9475efd57993395471ba0eb4175abfa5c0eb5674be3a37d1e WatchSource:0}: Error finding container 72835cd833dc59a9475efd57993395471ba0eb4175abfa5c0eb5674be3a37d1e: Status 404 returned error can't find the container with id 72835cd833dc59a9475efd57993395471ba0eb4175abfa5c0eb5674be3a37d1e
Nov 03 02:03:13.002089 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:13.002045    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w" event={"ID":"fbdeec42-7115-4361-ac6f-1874ffded23c","Type":"ContainerStarted","Data":"72835cd833dc59a9475efd57993395471ba0eb4175abfa5c0eb5674be3a37d1e"}
Nov 03 02:03:14.009412 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:14.009367    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w" event={"ID":"fbdeec42-7115-4361-ac6f-1874ffded23c","Type":"ContainerStarted","Data":"e4df71d557a63ffc96a498ba7c04c66861643797248eaed313af69878434127a"}
Nov 03 02:03:14.009820 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:14.009436    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"
Nov 03 02:03:14.117684 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:14.117641    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w" podStartSLOduration=3.294219698 podStartE2EDuration="4.117627409s" totalImagesPullingTime="823.407711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:14.115774344 +0000 UTC m=+721.130015710" watchObservedRunningTime="2026-11-03 02:03:14.117627409 +0000 UTC m=+721.131868774"
Nov 03 02:03:20.020882 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:20.020854    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"
Nov 03 02:03:25.608376 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:25.608337    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"]
Nov 03 02:03:25.614282 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:25.614256    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"
Nov 03 02:03:25.704800 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:25.704768    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a24e80-e0db-4bee-a1e8-eec710e6dd79-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4\" (UID: \"41a24e80-e0db-4bee-a1e8-eec710e6dd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"
Nov 03 02:03:25.704967 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:25.704934    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps27l\" (UniqueName: \"kubernetes.io/projected/41a24e80-e0db-4bee-a1e8-eec710e6dd79-kube-api-access-ps27l\") pod \"maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4\" (UID: \"41a24e80-e0db-4bee-a1e8-eec710e6dd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"
Nov 03 02:03:25.705051 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:25.704983    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a24e80-e0db-4bee-a1e8-eec710e6dd79-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4\" (UID: \"41a24e80-e0db-4bee-a1e8-eec710e6dd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"
Nov 03 02:03:25.705112 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:25.705089    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a24e80-e0db-4bee-a1e8-eec710e6dd79-metrics-certs\") pod \"maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4\" (UID: \"41a24e80-e0db-4bee-a1e8-eec710e6dd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"
Nov 03 02:03:25.709904 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:25.709880    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-f7d9bb-serving-cert\""
Nov 03 02:03:25.712653 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:25.712634    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"]
Nov 03 02:03:25.805663 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:25.805634    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps27l\" (UniqueName: \"kubernetes.io/projected/41a24e80-e0db-4bee-a1e8-eec710e6dd79-kube-api-access-ps27l\") pod \"maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4\" (UID: \"41a24e80-e0db-4bee-a1e8-eec710e6dd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"
Nov 03 02:03:25.805663 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:25.805672    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a24e80-e0db-4bee-a1e8-eec710e6dd79-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4\" (UID: \"41a24e80-e0db-4bee-a1e8-eec710e6dd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"
Nov 03 02:03:25.805952 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:25.805734    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a24e80-e0db-4bee-a1e8-eec710e6dd79-metrics-certs\") pod \"maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4\" (UID: \"41a24e80-e0db-4bee-a1e8-eec710e6dd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"
Nov 03 02:03:25.805952 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:25.805775    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a24e80-e0db-4bee-a1e8-eec710e6dd79-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4\" (UID: \"41a24e80-e0db-4bee-a1e8-eec710e6dd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"
Nov 03 02:03:25.805952 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:03:25.805858    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-serving-cert: secret "maas-api-e2e-worker-w0-f7d9bb-serving-cert" not found
Nov 03 02:03:25.805952 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:03:25.805940    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a24e80-e0db-4bee-a1e8-eec710e6dd79-maas-api-tls podName:41a24e80-e0db-4bee-a1e8-eec710e6dd79 nodeName:}" failed. No retries permitted until 2026-11-03 02:03:26.305920485 +0000 UTC m=+733.320161841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41a24e80-e0db-4bee-a1e8-eec710e6dd79-maas-api-tls") pod "maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4" (UID: "41a24e80-e0db-4bee-a1e8-eec710e6dd79") : secret "maas-api-e2e-worker-w0-f7d9bb-serving-cert" not found
Nov 03 02:03:25.806464 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:25.806446    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a24e80-e0db-4bee-a1e8-eec710e6dd79-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4\" (UID: \"41a24e80-e0db-4bee-a1e8-eec710e6dd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"
Nov 03 02:03:25.808241 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:25.808219    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a24e80-e0db-4bee-a1e8-eec710e6dd79-metrics-certs\") pod \"maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4\" (UID: \"41a24e80-e0db-4bee-a1e8-eec710e6dd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"
Nov 03 02:03:25.859282 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:25.859224    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps27l\" (UniqueName: \"kubernetes.io/projected/41a24e80-e0db-4bee-a1e8-eec710e6dd79-kube-api-access-ps27l\") pod \"maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4\" (UID: \"41a24e80-e0db-4bee-a1e8-eec710e6dd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"
Nov 03 02:03:26.310653 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:26.310623    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a24e80-e0db-4bee-a1e8-eec710e6dd79-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4\" (UID: \"41a24e80-e0db-4bee-a1e8-eec710e6dd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"
Nov 03 02:03:26.313037 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:26.313013    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a24e80-e0db-4bee-a1e8-eec710e6dd79-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4\" (UID: \"41a24e80-e0db-4bee-a1e8-eec710e6dd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"
Nov 03 02:03:26.527175 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:26.527144    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"
Nov 03 02:03:26.898068 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:26.897980    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"]
Nov 03 02:03:26.902394 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:03:26.902366    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a24e80_e0db_4bee_a1e8_eec710e6dd79.slice/crio-adbae7e70527c3bcdab0dc19cd05d39aed86fac00fc919aceea2adc31cece837 WatchSource:0}: Error finding container adbae7e70527c3bcdab0dc19cd05d39aed86fac00fc919aceea2adc31cece837: Status 404 returned error can't find the container with id adbae7e70527c3bcdab0dc19cd05d39aed86fac00fc919aceea2adc31cece837
Nov 03 02:03:27.079097 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:27.079065    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4" event={"ID":"41a24e80-e0db-4bee-a1e8-eec710e6dd79","Type":"ContainerStarted","Data":"adbae7e70527c3bcdab0dc19cd05d39aed86fac00fc919aceea2adc31cece837"}
Nov 03 02:03:28.086323 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:28.086284    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4" event={"ID":"41a24e80-e0db-4bee-a1e8-eec710e6dd79","Type":"ContainerStarted","Data":"2e26f39880f0cfa094282c101c7f6277d436fa3a97190b7d81b33a23e152f36c"}
Nov 03 02:03:28.086672 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:28.086367    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"
Nov 03 02:03:28.200467 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:28.200418    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4" podStartSLOduration=2.118153008 podStartE2EDuration="3.200404649s" totalImagesPullingTime="1.082251641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:28.197417296 +0000 UTC m=+735.211658662" watchObservedRunningTime="2026-11-03 02:03:28.200404649 +0000 UTC m=+735.214646014"
Nov 03 02:03:30.712342 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:03:30.712296    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:03:34.097030 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:34.097003    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"
Nov 03 02:03:38.498070 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.498038    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"]
Nov 03 02:03:38.502045 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.502028    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:38.516618 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.516593    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd71fd74-9581-4c83-8954-55462e1d4438-tls-certs\") pod \"simulator-w1-1bccbd-kserve-795b7bc77c-skmcd\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") " pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:38.516743 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.516634    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-dshm\") pod \"simulator-w1-1bccbd-kserve-795b7bc77c-skmcd\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") " pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:38.516743 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.516661    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-model-cache\") pod \"simulator-w1-1bccbd-kserve-795b7bc77c-skmcd\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") " pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:38.516842 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.516765    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-tmp-dir\") pod \"simulator-w1-1bccbd-kserve-795b7bc77c-skmcd\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") " pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:38.516842 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.516794    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-home\") pod \"simulator-w1-1bccbd-kserve-795b7bc77c-skmcd\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") " pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:38.516842 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.516827    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spg95\" (UniqueName: \"kubernetes.io/projected/fd71fd74-9581-4c83-8954-55462e1d4438-kube-api-access-spg95\") pod \"simulator-w1-1bccbd-kserve-795b7bc77c-skmcd\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") " pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:38.553040 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.553016    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1bccbd\"/\"openshift-service-ca.crt\""
Nov 03 02:03:38.599397 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.599376    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1bccbd\"/\"kube-root-ca.crt\""
Nov 03 02:03:38.599580 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.599563    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1bccbd\"/\"default-dockercfg-zdpps\""
Nov 03 02:03:38.599666 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.599592    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1bccbd\"/\"simulator-w1-1bccbd-kserve-self-signed-certs\""
Nov 03 02:03:38.601914 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.601894    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"]
Nov 03 02:03:38.618061 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.618030    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-tmp-dir\") pod \"simulator-w1-1bccbd-kserve-795b7bc77c-skmcd\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") " pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:38.618061 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.618066    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-home\") pod \"simulator-w1-1bccbd-kserve-795b7bc77c-skmcd\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") " pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:38.618217 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.618097    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spg95\" (UniqueName: \"kubernetes.io/projected/fd71fd74-9581-4c83-8954-55462e1d4438-kube-api-access-spg95\") pod \"simulator-w1-1bccbd-kserve-795b7bc77c-skmcd\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") " pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:38.618217 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.618161    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd71fd74-9581-4c83-8954-55462e1d4438-tls-certs\") pod \"simulator-w1-1bccbd-kserve-795b7bc77c-skmcd\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") " pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:38.618217 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.618179    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-dshm\") pod \"simulator-w1-1bccbd-kserve-795b7bc77c-skmcd\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") " pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:38.618379 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.618292    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-model-cache\") pod \"simulator-w1-1bccbd-kserve-795b7bc77c-skmcd\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") " pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:38.618462 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.618444    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-home\") pod \"simulator-w1-1bccbd-kserve-795b7bc77c-skmcd\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") " pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:38.618523 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.618505    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-tmp-dir\") pod \"simulator-w1-1bccbd-kserve-795b7bc77c-skmcd\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") " pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:38.618615 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.618600    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-model-cache\") pod \"simulator-w1-1bccbd-kserve-795b7bc77c-skmcd\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") " pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:38.620255 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.620239    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-dshm\") pod \"simulator-w1-1bccbd-kserve-795b7bc77c-skmcd\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") " pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:38.620510 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.620495    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd71fd74-9581-4c83-8954-55462e1d4438-tls-certs\") pod \"simulator-w1-1bccbd-kserve-795b7bc77c-skmcd\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") " pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:38.671438 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.671407    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spg95\" (UniqueName: \"kubernetes.io/projected/fd71fd74-9581-4c83-8954-55462e1d4438-kube-api-access-spg95\") pod \"simulator-w1-1bccbd-kserve-795b7bc77c-skmcd\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") " pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:38.813659 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:38.813627    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:39.193206 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:39.193135    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"]
Nov 03 02:03:39.198911 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:03:39.198885    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd71fd74_9581_4c83_8954_55462e1d4438.slice/crio-2ff2302c5181f74251bfd7c120d54d0455414fde005017ba1e1031ec492e5f16 WatchSource:0}: Error finding container 2ff2302c5181f74251bfd7c120d54d0455414fde005017ba1e1031ec492e5f16: Status 404 returned error can't find the container with id 2ff2302c5181f74251bfd7c120d54d0455414fde005017ba1e1031ec492e5f16
Nov 03 02:03:40.151375 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:40.151337    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd" event={"ID":"fd71fd74-9581-4c83-8954-55462e1d4438","Type":"ContainerStarted","Data":"0bbc2b454345e7a7ee8be0b9f2425976497a4293a9426a3fa2cd35a26a2b3ef0"}
Nov 03 02:03:40.151838 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:40.151384    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd" event={"ID":"fd71fd74-9581-4c83-8954-55462e1d4438","Type":"ContainerStarted","Data":"2ff2302c5181f74251bfd7c120d54d0455414fde005017ba1e1031ec492e5f16"}
Nov 03 02:03:40.259076 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:40.259029    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd" podStartSLOduration=2.259017452 podStartE2EDuration="2.259017452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:40.256386143 +0000 UTC m=+747.270627508" watchObservedRunningTime="2026-11-03 02:03:40.259017452 +0000 UTC m=+747.273258820"
Nov 03 02:03:40.814683 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:40.814653    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:48.814388 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:48.814351    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:48.827382 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:48.827355    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:03:49.212594 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:03:49.212517    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:04:00.715412 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:04:00.715332    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:04:30.715098 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:04:30.715060    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:05:00.713261 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:05:00.713213    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:05:30.713477 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:05:30.713378    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:06:00.713244 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:06:00.713203    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:06:13.197544 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:13.197513    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:06:13.198256 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:13.198234    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:06:29.966485 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:29.966452    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"]
Nov 03 02:06:29.970280 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:29.970263    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.044023 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.043995    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06335e60-e878-4ad3-9630-fabee17ef5d9-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.044122 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.044044    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.044122 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.044062    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.044233 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.044213    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.044271 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.044252    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.044303 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.044270    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzzqg\" (UniqueName: \"kubernetes.io/projected/06335e60-e878-4ad3-9630-fabee17ef5d9-kube-api-access-bzzqg\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.044381 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.044363    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.044413 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.044391    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.044489 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.044415    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.065253 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.065230    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c7d758-openshift-default-dockercfg-kst7d\""
Nov 03 02:06:30.070710 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.070672    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"]
Nov 03 02:06:30.145339 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.145309    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.145547 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.145351    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.145547 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.145375    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzzqg\" (UniqueName: \"kubernetes.io/projected/06335e60-e878-4ad3-9630-fabee17ef5d9-kube-api-access-bzzqg\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.145547 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.145403    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.145547 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.145429    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.145547 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.145459    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.145547 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.145493    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06335e60-e878-4ad3-9630-fabee17ef5d9-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.145904 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.145705    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.145904 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.145744    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.146010 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.145920    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.146236 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.146213    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.146818 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.146255    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.146818 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.146323    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06335e60-e878-4ad3-9630-fabee17ef5d9-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.146818 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.146169    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.147885 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.147861    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.148726 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.148683    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.199459 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.199426    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.199673 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.199647    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzzqg\" (UniqueName: \"kubernetes.io/projected/06335e60-e878-4ad3-9630-fabee17ef5d9-kube-api-access-bzzqg\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.283288 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.283259    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:30.714106 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.714027    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"]
Nov 03 02:06:30.715775 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:06:30.715686    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:06:30.720290 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:06:30.720264    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06335e60_e878_4ad3_9630_fabee17ef5d9.slice/crio-e121ceecff4c74daf716e7882d6f018d5d0698b4422feb534c9da94132e1ae0e WatchSource:0}: Error finding container e121ceecff4c74daf716e7882d6f018d5d0698b4422feb534c9da94132e1ae0e: Status 404 returned error can't find the container with id e121ceecff4c74daf716e7882d6f018d5d0698b4422feb534c9da94132e1ae0e
Nov 03 02:06:30.724716 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.723240    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:06:30.724716 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.723400    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:06:30.724716 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:30.723486    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:06:31.150098 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:31.150058    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w" event={"ID":"06335e60-e878-4ad3-9630-fabee17ef5d9","Type":"ContainerStarted","Data":"30b11792a4a028a500cfd2133f9ac904fce4278eca4ceff2993cb43f497b7212"}
Nov 03 02:06:31.150660 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:31.150636    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w" event={"ID":"06335e60-e878-4ad3-9630-fabee17ef5d9","Type":"ContainerStarted","Data":"e121ceecff4c74daf716e7882d6f018d5d0698b4422feb534c9da94132e1ae0e"}
Nov 03 02:06:31.265741 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:31.261101    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w" podStartSLOduration=2.261083958 podStartE2EDuration="2.261083958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:31.259345592 +0000 UTC m=+918.273586959" watchObservedRunningTime="2026-11-03 02:06:31.261083958 +0000 UTC m=+918.275325324"
Nov 03 02:06:31.285460 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:31.283735    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:31.286204 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:31.285881    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 02:06:31.286204 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:31.285941    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w" podUID="06335e60-e878-4ad3-9630-fabee17ef5d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 02:06:32.283754 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:32.283714    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 02:06:32.284245 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:32.283773    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w" podUID="06335e60-e878-4ad3-9630-fabee17ef5d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 02:06:33.283753 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:33.283716    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 02:06:33.283921 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:33.283767    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w" podUID="06335e60-e878-4ad3-9630-fabee17ef5d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 02:06:34.284520 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:34.284486    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 02:06:34.284993 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:34.284545    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w" podUID="06335e60-e878-4ad3-9630-fabee17ef5d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 02:06:35.284561 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:35.284525    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 02:06:35.284955 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:35.284579    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w" podUID="06335e60-e878-4ad3-9630-fabee17ef5d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 02:06:36.137458 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.137419    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"]
Nov 03 02:06:36.141564 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.141546    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"
Nov 03 02:06:36.191367 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.191330    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1dca5dd705c7\""
Nov 03 02:06:36.237866 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.237834    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 02:06:36.238111 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.237834    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 02:06:36.238111 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.237834    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 02:06:36.238283 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.237913    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 02:06:36.238283 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.237922    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-kf7p8\""
Nov 03 02:06:36.242306 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.242276    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"]
Nov 03 02:06:36.249723 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.249673    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84fc784f-4db7-430b-971b-6ab793ecc2aa-routing-overlay-volume\") pod \"payload-processing-external-model-65d88c7d45-hpcjv\" (UID: \"84fc784f-4db7-430b-971b-6ab793ecc2aa\") " pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"
Nov 03 02:06:36.249873 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.249753    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84fc784f-4db7-430b-971b-6ab793ecc2aa-provider-credentials\") pod \"payload-processing-external-model-65d88c7d45-hpcjv\" (UID: \"84fc784f-4db7-430b-971b-6ab793ecc2aa\") " pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"
Nov 03 02:06:36.249945 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.249898    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgzdc\" (UniqueName: \"kubernetes.io/projected/84fc784f-4db7-430b-971b-6ab793ecc2aa-kube-api-access-pgzdc\") pod \"payload-processing-external-model-65d88c7d45-hpcjv\" (UID: \"84fc784f-4db7-430b-971b-6ab793ecc2aa\") " pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"
Nov 03 02:06:36.250006 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.249954    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84fc784f-4db7-430b-971b-6ab793ecc2aa-plugins-config-volume\") pod \"payload-processing-external-model-65d88c7d45-hpcjv\" (UID: \"84fc784f-4db7-430b-971b-6ab793ecc2aa\") " pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"
Nov 03 02:06:36.284401 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.284371    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 03 02:06:36.284583 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.284432    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w" podUID="06335e60-e878-4ad3-9630-fabee17ef5d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 03 02:06:36.351375 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.351332    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgzdc\" (UniqueName: \"kubernetes.io/projected/84fc784f-4db7-430b-971b-6ab793ecc2aa-kube-api-access-pgzdc\") pod \"payload-processing-external-model-65d88c7d45-hpcjv\" (UID: \"84fc784f-4db7-430b-971b-6ab793ecc2aa\") " pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"
Nov 03 02:06:36.351560 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.351389    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84fc784f-4db7-430b-971b-6ab793ecc2aa-plugins-config-volume\") pod \"payload-processing-external-model-65d88c7d45-hpcjv\" (UID: \"84fc784f-4db7-430b-971b-6ab793ecc2aa\") " pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"
Nov 03 02:06:36.351560 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.351516    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84fc784f-4db7-430b-971b-6ab793ecc2aa-routing-overlay-volume\") pod \"payload-processing-external-model-65d88c7d45-hpcjv\" (UID: \"84fc784f-4db7-430b-971b-6ab793ecc2aa\") " pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"
Nov 03 02:06:36.351651 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.351571    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84fc784f-4db7-430b-971b-6ab793ecc2aa-provider-credentials\") pod \"payload-processing-external-model-65d88c7d45-hpcjv\" (UID: \"84fc784f-4db7-430b-971b-6ab793ecc2aa\") " pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"
Nov 03 02:06:36.352295 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.352270    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84fc784f-4db7-430b-971b-6ab793ecc2aa-routing-overlay-volume\") pod \"payload-processing-external-model-65d88c7d45-hpcjv\" (UID: \"84fc784f-4db7-430b-971b-6ab793ecc2aa\") " pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"
Nov 03 02:06:36.352363 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.352348    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84fc784f-4db7-430b-971b-6ab793ecc2aa-plugins-config-volume\") pod \"payload-processing-external-model-65d88c7d45-hpcjv\" (UID: \"84fc784f-4db7-430b-971b-6ab793ecc2aa\") " pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"
Nov 03 02:06:36.354206 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.354185    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84fc784f-4db7-430b-971b-6ab793ecc2aa-provider-credentials\") pod \"payload-processing-external-model-65d88c7d45-hpcjv\" (UID: \"84fc784f-4db7-430b-971b-6ab793ecc2aa\") " pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"
Nov 03 02:06:36.424154 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.424078    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgzdc\" (UniqueName: \"kubernetes.io/projected/84fc784f-4db7-430b-971b-6ab793ecc2aa-kube-api-access-pgzdc\") pod \"payload-processing-external-model-65d88c7d45-hpcjv\" (UID: \"84fc784f-4db7-430b-971b-6ab793ecc2aa\") " pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"
Nov 03 02:06:36.456462 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.456433    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"
Nov 03 02:06:36.856382 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:36.856342    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"]
Nov 03 02:06:36.863796 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:06:36.863761    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84fc784f_4db7_430b_971b_6ab793ecc2aa.slice/crio-7b283695a16162c2afe7cc55e86222efa05c063914a3f1db0ea731de1591e448 WatchSource:0}: Error finding container 7b283695a16162c2afe7cc55e86222efa05c063914a3f1db0ea731de1591e448: Status 404 returned error can't find the container with id 7b283695a16162c2afe7cc55e86222efa05c063914a3f1db0ea731de1591e448
Nov 03 02:06:37.204217 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:37.204114    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv" event={"ID":"84fc784f-4db7-430b-971b-6ab793ecc2aa","Type":"ContainerStarted","Data":"9617866d0799f6220159eddea0bf50259e1e3df1bcad189373d6ab04ba155ca2"}
Nov 03 02:06:37.204217 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:37.204182    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv" event={"ID":"84fc784f-4db7-430b-971b-6ab793ecc2aa","Type":"ContainerStarted","Data":"7b283695a16162c2afe7cc55e86222efa05c063914a3f1db0ea731de1591e448"}
Nov 03 02:06:37.204428 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:37.204268    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"
Nov 03 02:06:37.311996 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:37.311928    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv" podStartSLOduration=1.311907236 podStartE2EDuration="1.311907236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:37.310580392 +0000 UTC m=+924.324821770" watchObservedRunningTime="2026-11-03 02:06:37.311907236 +0000 UTC m=+924.326148604"
Nov 03 02:06:37.492260 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:37.492163    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:37.492586 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:37.492564    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:37.493626 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:37.493603    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:06:48.213529 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:48.213495    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv" podUID="84fc784f-4db7-430b-971b-6ab793ecc2aa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:06:50.329034 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.329001    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"]
Nov 03 02:06:50.333139 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.333117    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.412074 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.412043    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9m2r\" (UniqueName: \"kubernetes.io/projected/af839e9e-267b-4c27-b8cc-4e4e591580e3-kube-api-access-t9m2r\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.412226 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.412091    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.412226 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.412143    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.412226 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.412168    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af839e9e-267b-4c27-b8cc-4e4e591580e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.412226 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.412201    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.412364 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.412229    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.412364 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.412332    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.412431 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.412363    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.412431 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.412402    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.432345 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.432313    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"]
Nov 03 02:06:50.513482 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.513449    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.513482 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.513485    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.513766 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.513508    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af839e9e-267b-4c27-b8cc-4e4e591580e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.513766 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.513539    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.513766 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.513567    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.513766 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.513593    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.513766 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.513617    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.513766 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.513667    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.514088 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.513789    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9m2r\" (UniqueName: \"kubernetes.io/projected/af839e9e-267b-4c27-b8cc-4e4e591580e3-kube-api-access-t9m2r\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.514088 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.513971    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.514088 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.514067    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.514258 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.514180    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.514258 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.514204    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.514563 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.514541    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af839e9e-267b-4c27-b8cc-4e4e591580e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.515861 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.515841    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.516127 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.516106    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.566508 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.566475    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.566508 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.566500    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9m2r\" (UniqueName: \"kubernetes.io/projected/af839e9e-267b-4c27-b8cc-4e4e591580e3-kube-api-access-t9m2r\") pod \"maas-default-gateway-openshift-default-5bc884dd87-h4htz\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:50.646745 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:50.646659    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:51.073061 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:51.073027    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"]
Nov 03 02:06:51.079020 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:06:51.078994    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf839e9e_267b_4c27_b8cc_4e4e591580e3.slice/crio-ce6e7bbacdf87a5eef93bc6a9c85af098498eb3fccdcdccbab21e9dcc1c63c2f WatchSource:0}: Error finding container ce6e7bbacdf87a5eef93bc6a9c85af098498eb3fccdcdccbab21e9dcc1c63c2f: Status 404 returned error can't find the container with id ce6e7bbacdf87a5eef93bc6a9c85af098498eb3fccdcdccbab21e9dcc1c63c2f
Nov 03 02:06:51.081402 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:51.081370    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:06:51.081458 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:51.081434    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:06:51.081497 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:51.081464    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:06:51.294052 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:51.294020    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz" event={"ID":"af839e9e-267b-4c27-b8cc-4e4e591580e3","Type":"ContainerStarted","Data":"2d701cb3cf9e41322d8b8d01265d13fdc81ef21fa27aa6d2f3423d7c0e6643df"}
Nov 03 02:06:51.294223 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:51.294072    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz" event={"ID":"af839e9e-267b-4c27-b8cc-4e4e591580e3","Type":"ContainerStarted","Data":"ce6e7bbacdf87a5eef93bc6a9c85af098498eb3fccdcdccbab21e9dcc1c63c2f"}
Nov 03 02:06:51.406871 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:51.406762    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz" podStartSLOduration=1.406737434 podStartE2EDuration="1.406737434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:51.401623707 +0000 UTC m=+938.415865138" watchObservedRunningTime="2026-11-03 02:06:51.406737434 +0000 UTC m=+938.420978818"
Nov 03 02:06:51.652720 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:51.647810    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:52.648601 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:52.648566    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-h4htz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 02:06:52.649055 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:52.648621    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz" podUID="af839e9e-267b-4c27-b8cc-4e4e591580e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 02:06:53.647042 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:53.647006    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-h4htz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 02:06:53.647229 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:53.647059    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz" podUID="af839e9e-267b-4c27-b8cc-4e4e591580e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 02:06:54.647175 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:54.647140    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-h4htz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 02:06:54.647553 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:54.647196    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz" podUID="af839e9e-267b-4c27-b8cc-4e4e591580e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 02:06:55.647757 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:55.647687    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-h4htz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 02:06:55.648179 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:55.647779    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz" podUID="af839e9e-267b-4c27-b8cc-4e4e591580e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 02:06:56.647614 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:56.647577    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-h4htz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 02:06:56.647817 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:56.647634    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz" podUID="af839e9e-267b-4c27-b8cc-4e4e591580e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 02:06:57.647606 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:57.647572    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-h4htz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 02:06:57.647802 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:57.647642    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz" podUID="af839e9e-267b-4c27-b8cc-4e4e591580e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 02:06:58.211892 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:58.211851    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv" podUID="84fc784f-4db7-430b-971b-6ab793ecc2aa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:06:58.648039 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:58.648002    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-h4htz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 03 02:06:58.648248 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:58.648059    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz" podUID="af839e9e-267b-4c27-b8cc-4e4e591580e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 03 02:06:59.694505 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:59.694468    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:59.695066 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:59.695045    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:06:59.695907 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:06:59.695890    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:07:00.713749 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:07:00.713689    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:07:08.211829 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:08.211793    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv" podUID="84fc784f-4db7-430b-971b-6ab793ecc2aa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:07:18.211636 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:18.211591    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv" podUID="84fc784f-4db7-430b-971b-6ab793ecc2aa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:07:21.072370 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:21.072338    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"]
Nov 03 02:07:21.073631 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:21.073602    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv" podUID="84fc784f-4db7-430b-971b-6ab793ecc2aa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:07:24.003809 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.003767    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"]
Nov 03 02:07:24.009184 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.009160    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:24.065211 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.065173    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"]
Nov 03 02:07:24.074410 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.074375    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-home\") pod \"distinct-w4-c7d758-kserve-d4c9f769f-6v4td\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:24.074570 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.074424    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-dshm\") pod \"distinct-w4-c7d758-kserve-d4c9f769f-6v4td\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:24.074570 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.074454    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-model-cache\") pod \"distinct-w4-c7d758-kserve-d4c9f769f-6v4td\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:24.074661 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.074563    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-tmp-dir\") pod \"distinct-w4-c7d758-kserve-d4c9f769f-6v4td\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:24.074661 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.074606    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4vvx\" (UniqueName: \"kubernetes.io/projected/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-kube-api-access-g4vvx\") pod \"distinct-w4-c7d758-kserve-d4c9f769f-6v4td\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:24.074819 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.074714    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-tls-certs\") pod \"distinct-w4-c7d758-kserve-d4c9f769f-6v4td\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:24.106767 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.106739    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c7d758\"/\"kube-root-ca.crt\""
Nov 03 02:07:24.106767 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.106764    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c7d758\"/\"openshift-service-ca.crt\""
Nov 03 02:07:24.106996 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.106765    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c7d758\"/\"default-dockercfg-7k4bc\""
Nov 03 02:07:24.106996 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.106943    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c7d758\"/\"distinct-w4-c7d758-kserve-self-signed-certs\""
Nov 03 02:07:24.175931 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.175892    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-home\") pod \"distinct-w4-c7d758-kserve-d4c9f769f-6v4td\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:24.176140 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.175938    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-dshm\") pod \"distinct-w4-c7d758-kserve-d4c9f769f-6v4td\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:24.176140 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.175969    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-model-cache\") pod \"distinct-w4-c7d758-kserve-d4c9f769f-6v4td\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:24.176140 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.176032    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-tmp-dir\") pod \"distinct-w4-c7d758-kserve-d4c9f769f-6v4td\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:24.176140 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.176091    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4vvx\" (UniqueName: \"kubernetes.io/projected/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-kube-api-access-g4vvx\") pod \"distinct-w4-c7d758-kserve-d4c9f769f-6v4td\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:24.176371 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.176144    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-tls-certs\") pod \"distinct-w4-c7d758-kserve-d4c9f769f-6v4td\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:24.176533 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.176510    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-model-cache\") pod \"distinct-w4-c7d758-kserve-d4c9f769f-6v4td\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:24.176581 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.176521    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-home\") pod \"distinct-w4-c7d758-kserve-d4c9f769f-6v4td\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:24.176725 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.176670    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-tmp-dir\") pod \"distinct-w4-c7d758-kserve-d4c9f769f-6v4td\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:24.178555 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.178535    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-dshm\") pod \"distinct-w4-c7d758-kserve-d4c9f769f-6v4td\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:24.179006 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.178986    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-tls-certs\") pod \"distinct-w4-c7d758-kserve-d4c9f769f-6v4td\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:24.229547 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.229515    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4vvx\" (UniqueName: \"kubernetes.io/projected/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-kube-api-access-g4vvx\") pod \"distinct-w4-c7d758-kserve-d4c9f769f-6v4td\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") " pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:24.321315 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.321230    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:24.696707 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:24.696606    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"]
Nov 03 02:07:24.703042 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:07:24.703014    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f39ea17_c4d7_4de2_bb69_c7d3012c5f77.slice/crio-1282d4636f6214746fb2cdc6bbc0f230147777bc94892eb7a0055a34b1636260 WatchSource:0}: Error finding container 1282d4636f6214746fb2cdc6bbc0f230147777bc94892eb7a0055a34b1636260: Status 404 returned error can't find the container with id 1282d4636f6214746fb2cdc6bbc0f230147777bc94892eb7a0055a34b1636260
Nov 03 02:07:25.511738 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:25.511682    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td" event={"ID":"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77","Type":"ContainerStarted","Data":"12bc1b6324d761be75c7e809d95ad5c8cabc8db4ab1d80fa7aab434181cc09b8"}
Nov 03 02:07:25.511738 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:25.511742    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td" event={"ID":"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77","Type":"ContainerStarted","Data":"1282d4636f6214746fb2cdc6bbc0f230147777bc94892eb7a0055a34b1636260"}
Nov 03 02:07:25.627360 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:25.627294    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td" podStartSLOduration=2.627274502 podStartE2EDuration="2.627274502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:07:25.621598008 +0000 UTC m=+972.635839378" watchObservedRunningTime="2026-11-03 02:07:25.627274502 +0000 UTC m=+972.641515869"
Nov 03 02:07:26.073008 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.072921    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv" podUID="84fc784f-4db7-430b-971b-6ab793ecc2aa" containerName="payload-processing" containerID="cri-o://9617866d0799f6220159eddea0bf50259e1e3df1bcad189373d6ab04ba155ca2" gracePeriod=25
Nov 03 02:07:26.322054 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.322026    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:26.344147 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.344118    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"
Nov 03 02:07:26.399029 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.398995    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84fc784f-4db7-430b-971b-6ab793ecc2aa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84fc784f-4db7-430b-971b-6ab793ecc2aa-routing-overlay-volume\") pod \"84fc784f-4db7-430b-971b-6ab793ecc2aa\" (UID: \"84fc784f-4db7-430b-971b-6ab793ecc2aa\") "
Nov 03 02:07:26.399228 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.399094    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84fc784f-4db7-430b-971b-6ab793ecc2aa-kube-api-access-pgzdc\" (UniqueName: \"kubernetes.io/projected/84fc784f-4db7-430b-971b-6ab793ecc2aa-kube-api-access-pgzdc\") pod \"84fc784f-4db7-430b-971b-6ab793ecc2aa\" (UID: \"84fc784f-4db7-430b-971b-6ab793ecc2aa\") "
Nov 03 02:07:26.399228 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.399127    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84fc784f-4db7-430b-971b-6ab793ecc2aa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/84fc784f-4db7-430b-971b-6ab793ecc2aa-provider-credentials\") pod \"84fc784f-4db7-430b-971b-6ab793ecc2aa\" (UID: \"84fc784f-4db7-430b-971b-6ab793ecc2aa\") "
Nov 03 02:07:26.399228 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.399162    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84fc784f-4db7-430b-971b-6ab793ecc2aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84fc784f-4db7-430b-971b-6ab793ecc2aa-plugins-config-volume\") pod \"84fc784f-4db7-430b-971b-6ab793ecc2aa\" (UID: \"84fc784f-4db7-430b-971b-6ab793ecc2aa\") "
Nov 03 02:07:26.399425 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.399382    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84fc784f-4db7-430b-971b-6ab793ecc2aa-routing-overlay-volume" pod "84fc784f-4db7-430b-971b-6ab793ecc2aa" (UID: "84fc784f-4db7-430b-971b-6ab793ecc2aa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:07:26.399589 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.399569    5005 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84fc784f-4db7-430b-971b-6ab793ecc2aa-routing-overlay-volume\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:07:26.399673 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.399601    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84fc784f-4db7-430b-971b-6ab793ecc2aa-plugins-config-volume" pod "84fc784f-4db7-430b-971b-6ab793ecc2aa" (UID: "84fc784f-4db7-430b-971b-6ab793ecc2aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:07:26.401449 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.401427    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84fc784f-4db7-430b-971b-6ab793ecc2aa-provider-credentials" pod "84fc784f-4db7-430b-971b-6ab793ecc2aa" (UID: "84fc784f-4db7-430b-971b-6ab793ecc2aa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:07:26.401511 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.401448    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84fc784f-4db7-430b-971b-6ab793ecc2aa-kube-api-access-pgzdc" pod "84fc784f-4db7-430b-971b-6ab793ecc2aa" (UID: "84fc784f-4db7-430b-971b-6ab793ecc2aa"). InnerVolumeSpecName "kube-api-access-pgzdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:07:26.501254 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.501213    5005 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84fc784f-4db7-430b-971b-6ab793ecc2aa-provider-credentials\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:07:26.501254 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.501253    5005 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84fc784f-4db7-430b-971b-6ab793ecc2aa-plugins-config-volume\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:07:26.501529 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.501268    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgzdc\" (UniqueName: \"kubernetes.io/projected/84fc784f-4db7-430b-971b-6ab793ecc2aa-kube-api-access-pgzdc\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:07:26.519246 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.519209    5005 generic.go:356] "Generic (PLEG): container finished" podID="84fc784f-4db7-430b-971b-6ab793ecc2aa" containerID="9617866d0799f6220159eddea0bf50259e1e3df1bcad189373d6ab04ba155ca2" exitCode=0
Nov 03 02:07:26.519755 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.519261    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"
Nov 03 02:07:26.519755 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.519290    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv" event={"ID":"84fc784f-4db7-430b-971b-6ab793ecc2aa","Type":"ContainerDied","Data":"9617866d0799f6220159eddea0bf50259e1e3df1bcad189373d6ab04ba155ca2"}
Nov 03 02:07:26.519755 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.519374    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv" event={"ID":"84fc784f-4db7-430b-971b-6ab793ecc2aa","Type":"ContainerDied","Data":"7b283695a16162c2afe7cc55e86222efa05c063914a3f1db0ea731de1591e448"}
Nov 03 02:07:26.519755 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.519397    5005 scope.go:122] "RemoveContainer" containerID="9617866d0799f6220159eddea0bf50259e1e3df1bcad189373d6ab04ba155ca2"
Nov 03 02:07:26.534074 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.534050    5005 scope.go:122] "RemoveContainer" containerID="9617866d0799f6220159eddea0bf50259e1e3df1bcad189373d6ab04ba155ca2"
Nov 03 02:07:26.534405 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:07:26.534383    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9617866d0799f6220159eddea0bf50259e1e3df1bcad189373d6ab04ba155ca2\": container with ID starting with 9617866d0799f6220159eddea0bf50259e1e3df1bcad189373d6ab04ba155ca2 not found: ID does not exist" containerID="9617866d0799f6220159eddea0bf50259e1e3df1bcad189373d6ab04ba155ca2"
Nov 03 02:07:26.534510 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.534413    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9617866d0799f6220159eddea0bf50259e1e3df1bcad189373d6ab04ba155ca2"} err="failed to get container status \"9617866d0799f6220159eddea0bf50259e1e3df1bcad189373d6ab04ba155ca2\": rpc error: code = NotFound desc = could not find container \"9617866d0799f6220159eddea0bf50259e1e3df1bcad189373d6ab04ba155ca2\": container with ID starting with 9617866d0799f6220159eddea0bf50259e1e3df1bcad189373d6ab04ba155ca2 not found: ID does not exist"
Nov 03 02:07:26.672815 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.672741    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"]
Nov 03 02:07:26.676758 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:26.676727    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-65d88c7d45-hpcjv"]
Nov 03 02:07:27.112822 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:27.112784    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84fc784f-4db7-430b-971b-6ab793ecc2aa" path="/var/lib/kubelet/pods/84fc784f-4db7-430b-971b-6ab793ecc2aa/volumes"
Nov 03 02:07:30.713356 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:07:30.713321    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:07:34.321575 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:34.321537    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:34.334527 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:34.334498    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:34.604031 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:34.603955    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:07:48.412017 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.411977    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"]
Nov 03 02:07:48.412560 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.412543    5005 cpu_manager.go:414] "containerMap: removing container" podUID="84fc784f-4db7-430b-971b-6ab793ecc2aa" containerName="payload-processing"
Nov 03 02:07:48.412603 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.412561    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84fc784f-4db7-430b-971b-6ab793ecc2aa" containerName="payload-processing"
Nov 03 02:07:48.412724 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.412682    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="84fc784f-4db7-430b-971b-6ab793ecc2aa" containerName="payload-processing"
Nov 03 02:07:48.416306 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.416288    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"
Nov 03 02:07:48.468427 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.468394    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-06f459175dc6\""
Nov 03 02:07:48.471979 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.471947    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e33c474-783f-4077-9667-d405119e45f2-routing-overlay-volume\") pod \"payload-processing-external-model-664f597bf6-98v2w\" (UID: \"7e33c474-783f-4077-9667-d405119e45f2\") " pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"
Nov 03 02:07:48.472182 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.472137    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e33c474-783f-4077-9667-d405119e45f2-provider-credentials\") pod \"payload-processing-external-model-664f597bf6-98v2w\" (UID: \"7e33c474-783f-4077-9667-d405119e45f2\") " pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"
Nov 03 02:07:48.472182 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.472162    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghqrs\" (UniqueName: \"kubernetes.io/projected/7e33c474-783f-4077-9667-d405119e45f2-kube-api-access-ghqrs\") pod \"payload-processing-external-model-664f597bf6-98v2w\" (UID: \"7e33c474-783f-4077-9667-d405119e45f2\") " pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"
Nov 03 02:07:48.472293 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.472249    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e33c474-783f-4077-9667-d405119e45f2-plugins-config-volume\") pod \"payload-processing-external-model-664f597bf6-98v2w\" (UID: \"7e33c474-783f-4077-9667-d405119e45f2\") " pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"
Nov 03 02:07:48.514882 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.514845    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-kf7p8\""
Nov 03 02:07:48.515076 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.514845    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-e895c513fa82\""
Nov 03 02:07:48.515076 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.514912    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 02:07:48.515076 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.514860    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-dc8a8fe9a9e8\""
Nov 03 02:07:48.515202 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.515085    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 02:07:48.515202 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.515103    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 02:07:48.515202 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.515122    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 02:07:48.518274 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.518250    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"]
Nov 03 02:07:48.573224 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.573193    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e33c474-783f-4077-9667-d405119e45f2-provider-credentials\") pod \"payload-processing-external-model-664f597bf6-98v2w\" (UID: \"7e33c474-783f-4077-9667-d405119e45f2\") " pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"
Nov 03 02:07:48.573224 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.573226    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghqrs\" (UniqueName: \"kubernetes.io/projected/7e33c474-783f-4077-9667-d405119e45f2-kube-api-access-ghqrs\") pod \"payload-processing-external-model-664f597bf6-98v2w\" (UID: \"7e33c474-783f-4077-9667-d405119e45f2\") " pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"
Nov 03 02:07:48.573524 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.573271    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e33c474-783f-4077-9667-d405119e45f2-plugins-config-volume\") pod \"payload-processing-external-model-664f597bf6-98v2w\" (UID: \"7e33c474-783f-4077-9667-d405119e45f2\") " pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"
Nov 03 02:07:48.573524 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.573310    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e33c474-783f-4077-9667-d405119e45f2-routing-overlay-volume\") pod \"payload-processing-external-model-664f597bf6-98v2w\" (UID: \"7e33c474-783f-4077-9667-d405119e45f2\") " pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"
Nov 03 02:07:48.574156 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.574126    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e33c474-783f-4077-9667-d405119e45f2-routing-overlay-volume\") pod \"payload-processing-external-model-664f597bf6-98v2w\" (UID: \"7e33c474-783f-4077-9667-d405119e45f2\") " pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"
Nov 03 02:07:48.574284 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.574196    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e33c474-783f-4077-9667-d405119e45f2-plugins-config-volume\") pod \"payload-processing-external-model-664f597bf6-98v2w\" (UID: \"7e33c474-783f-4077-9667-d405119e45f2\") " pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"
Nov 03 02:07:48.575997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.575975    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e33c474-783f-4077-9667-d405119e45f2-provider-credentials\") pod \"payload-processing-external-model-664f597bf6-98v2w\" (UID: \"7e33c474-783f-4077-9667-d405119e45f2\") " pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"
Nov 03 02:07:48.626015 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.625977    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghqrs\" (UniqueName: \"kubernetes.io/projected/7e33c474-783f-4077-9667-d405119e45f2-kube-api-access-ghqrs\") pod \"payload-processing-external-model-664f597bf6-98v2w\" (UID: \"7e33c474-783f-4077-9667-d405119e45f2\") " pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"
Nov 03 02:07:48.730267 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:48.730177    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"
Nov 03 02:07:49.108852 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:49.108829    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"]
Nov 03 02:07:49.111335 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:07:49.111308    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e33c474_783f_4077_9667_d405119e45f2.slice/crio-ef996ae0c6721774d8e069094cfe0c1b08536488301f76d60161e32f2c5a0f6d WatchSource:0}: Error finding container ef996ae0c6721774d8e069094cfe0c1b08536488301f76d60161e32f2c5a0f6d: Status 404 returned error can't find the container with id ef996ae0c6721774d8e069094cfe0c1b08536488301f76d60161e32f2c5a0f6d
Nov 03 02:07:49.681512 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:49.681475    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w" event={"ID":"7e33c474-783f-4077-9667-d405119e45f2","Type":"ContainerStarted","Data":"18cc856fd1be14e4b6d5ec8bdd16ecff317da69308c5300ddda6e3dde831e411"}
Nov 03 02:07:49.681948 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:49.681539    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"
Nov 03 02:07:49.681948 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:49.681556    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w" event={"ID":"7e33c474-783f-4077-9667-d405119e45f2","Type":"ContainerStarted","Data":"ef996ae0c6721774d8e069094cfe0c1b08536488301f76d60161e32f2c5a0f6d"}
Nov 03 02:07:49.790421 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:07:49.790363    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w" podStartSLOduration=1.790345896 podStartE2EDuration="1.790345896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:07:49.787069209 +0000 UTC m=+996.801310574" watchObservedRunningTime="2026-11-03 02:07:49.790345896 +0000 UTC m=+996.804587293"
Nov 03 02:08:00.633758 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.633725    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"]
Nov 03 02:08:00.637754 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.637728    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.692305 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.692275    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"
Nov 03 02:08:00.713392 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:08:00.713356    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:08:00.735917 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.735887    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"]
Nov 03 02:08:00.807515 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.807479    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.807722 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.807607    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgzqv\" (UniqueName: \"kubernetes.io/projected/7d6ce282-686b-432c-bb78-7361aecf6a05-kube-api-access-dgzqv\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.807722 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.807669    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.807816 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.807775    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6ce282-686b-432c-bb78-7361aecf6a05-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.807816 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.807807    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.807939 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.807913    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.808050 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.808027    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.808108 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.808094    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.808151 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.808143    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.909457 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.909377    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.909457 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.909421    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6ce282-686b-432c-bb78-7361aecf6a05-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.909457 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.909441    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.909776 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.909563    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.909776 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.909622    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.909776 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.909689    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.909932 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.909794    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.909932 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.909831    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.909932 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.909872    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.910099 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.909928    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.910099 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.910072    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.910226 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.909933    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgzqv\" (UniqueName: \"kubernetes.io/projected/7d6ce282-686b-432c-bb78-7361aecf6a05-kube-api-access-dgzqv\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.910226 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.910129    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.910355 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.910309    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6ce282-686b-432c-bb78-7361aecf6a05-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.912063 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.912042    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.912233 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.912214    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.963193 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.963164    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:00.963357 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:00.963302    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgzqv\" (UniqueName: \"kubernetes.io/projected/7d6ce282-686b-432c-bb78-7361aecf6a05-kube-api-access-dgzqv\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:01.251881 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:01.251855    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:01.674974 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:01.674888    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"]
Nov 03 02:08:01.680493 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:08:01.680467    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6ce282_686b_432c_bb78_7361aecf6a05.slice/crio-6d38fce96a0016bea98d055677aaea324384cd0677c9d436b77135ad88ab3dd8 WatchSource:0}: Error finding container 6d38fce96a0016bea98d055677aaea324384cd0677c9d436b77135ad88ab3dd8: Status 404 returned error can't find the container with id 6d38fce96a0016bea98d055677aaea324384cd0677c9d436b77135ad88ab3dd8
Nov 03 02:08:01.682669 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:01.682635    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:08:01.682775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:01.682716    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:08:01.682775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:01.682746    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:08:01.751611 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:01.751584    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"]
Nov 03 02:08:01.756144 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:01.756121    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:01.756811 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:01.756781    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh" event={"ID":"7d6ce282-686b-432c-bb78-7361aecf6a05","Type":"ContainerStarted","Data":"e485576da0fb9c8e9165dffb4cbb13f9f2c0aa538dfac85ea87b19822dcd78b5"}
Nov 03 02:08:01.756910 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:01.756838    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh" event={"ID":"7d6ce282-686b-432c-bb78-7361aecf6a05","Type":"ContainerStarted","Data":"6d38fce96a0016bea98d055677aaea324384cd0677c9d436b77135ad88ab3dd8"}
Nov 03 02:08:01.806686 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:01.806645    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-1bccbd-openshift-default-dockercfg-c57w7\""
Nov 03 02:08:01.838336 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:01.838304    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"]
Nov 03 02:08:01.922990 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:01.922955    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:01.923191 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:01.923097    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrzrz\" (UniqueName: \"kubernetes.io/projected/00145046-700c-436b-a729-4332d8935371-kube-api-access-mrzrz\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:01.923191 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:01.923139    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:01.923331 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:01.923303    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:01.923391 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:01.923346    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00145046-700c-436b-a729-4332d8935371-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:01.923391 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:01.923373    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00145046-700c-436b-a729-4332d8935371-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:01.923499 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:01.923473    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:01.923580 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:01.923558    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:01.923713 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:01.923621    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00145046-700c-436b-a729-4332d8935371-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:01.987557 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:01.976775    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh" podStartSLOduration=1.976755561 podStartE2EDuration="1.976755561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:08:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:08:01.973408554 +0000 UTC m=+1008.987649920" watchObservedRunningTime="2026-11-03 02:08:01.976755561 +0000 UTC m=+1008.990996928"
Nov 03 02:08:02.025124 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.025085    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:02.025301 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.025132    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00145046-700c-436b-a729-4332d8935371-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:02.025301 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.025164    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00145046-700c-436b-a729-4332d8935371-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:02.025301 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.025217    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:02.025301 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.025256    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:02.025301 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.025283    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00145046-700c-436b-a729-4332d8935371-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:02.025557 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.025328    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:02.025557 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.025409    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrzrz\" (UniqueName: \"kubernetes.io/projected/00145046-700c-436b-a729-4332d8935371-kube-api-access-mrzrz\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:02.025557 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.025462    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:02.025736 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.025600    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:02.026401 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.026373    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:02.026772 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.026407    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:02.026772 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.026464    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:02.026955 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.026863    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00145046-700c-436b-a729-4332d8935371-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:02.028959 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.028933    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:02.029073 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.028992    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00145046-700c-436b-a729-4332d8935371-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:02.078814 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.078778    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrzrz\" (UniqueName: \"kubernetes.io/projected/00145046-700c-436b-a729-4332d8935371-kube-api-access-mrzrz\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:02.079093 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.079068    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00145046-700c-436b-a729-4332d8935371-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl\" (UID: \"00145046-700c-436b-a729-4332d8935371\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:02.253050 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.253025    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:02.254769 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.254747    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 02:08:02.254825 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.254797    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh" podUID="7d6ce282-686b-432c-bb78-7361aecf6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 02:08:02.370858 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.370822    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:02.824185 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.824139    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"]
Nov 03 02:08:02.826279 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:08:02.826252    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00145046_700c_436b_a729_4332d8935371.slice/crio-f03acb1ecd57fb3bb64af461b0c869770a97aebaed76d77f93d9bc470f70f00c WatchSource:0}: Error finding container f03acb1ecd57fb3bb64af461b0c869770a97aebaed76d77f93d9bc470f70f00c: Status 404 returned error can't find the container with id f03acb1ecd57fb3bb64af461b0c869770a97aebaed76d77f93d9bc470f70f00c
Nov 03 02:08:02.828708 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.828661    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:08:02.828805 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.828748    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:08:02.828805 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:02.828785    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:08:03.252560 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:03.252530    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 02:08:03.252731 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:03.252582    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh" podUID="7d6ce282-686b-432c-bb78-7361aecf6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 02:08:03.770264 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:03.770222    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl" event={"ID":"00145046-700c-436b-a729-4332d8935371","Type":"ContainerStarted","Data":"24ec2c025816d2a18e1595dfd26e3cb3a74ab4e2ca6e8b525ba9131025414df0"}
Nov 03 02:08:03.770478 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:03.770284    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl" event={"ID":"00145046-700c-436b-a729-4332d8935371","Type":"ContainerStarted","Data":"f03acb1ecd57fb3bb64af461b0c869770a97aebaed76d77f93d9bc470f70f00c"}
Nov 03 02:08:03.981924 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:03.981851    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl" podStartSLOduration=2.9818323920000003 podStartE2EDuration="2.981832392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:08:03.93225339 +0000 UTC m=+1010.946494761" watchObservedRunningTime="2026-11-03 02:08:03.981832392 +0000 UTC m=+1010.996073761"
Nov 03 02:08:04.253064 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:04.253037    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 02:08:04.253227 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:04.253089    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh" podUID="7d6ce282-686b-432c-bb78-7361aecf6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 02:08:04.371319 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:04.371288    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:04.372637 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:04.372616    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 02:08:04.372801 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:04.372774    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl" podUID="00145046-700c-436b-a729-4332d8935371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 02:08:05.253831 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:05.253791    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 02:08:05.254449 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:05.254412    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh" podUID="7d6ce282-686b-432c-bb78-7361aecf6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 02:08:05.371292 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:05.371221    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 02:08:05.371467 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:05.371357    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl" podUID="00145046-700c-436b-a729-4332d8935371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 02:08:06.255913 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:06.255874    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 02:08:06.256465 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:06.256434    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh" podUID="7d6ce282-686b-432c-bb78-7361aecf6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 02:08:06.372220 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:06.372183    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 02:08:06.372552 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:06.372498    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl" podUID="00145046-700c-436b-a729-4332d8935371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 02:08:07.267088 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:07.267049    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 02:08:07.267668 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:07.267631    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh" podUID="7d6ce282-686b-432c-bb78-7361aecf6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 02:08:07.372299 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:07.372154    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 02:08:07.372299 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:07.372220    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl" podUID="00145046-700c-436b-a729-4332d8935371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 02:08:08.254717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:08.254091    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 02:08:08.254717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:08.254159    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh" podUID="7d6ce282-686b-432c-bb78-7361aecf6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 02:08:08.389866 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:08.389822    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 02:08:08.390468 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:08.390434    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl" podUID="00145046-700c-436b-a729-4332d8935371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 02:08:09.255101 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:09.255059    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 02:08:09.255384 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:09.255354    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh" podUID="7d6ce282-686b-432c-bb78-7361aecf6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 02:08:09.384134 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:09.384100    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 02:08:09.384413 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:09.384387    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl" podUID="00145046-700c-436b-a729-4332d8935371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 02:08:10.257867 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:10.257840    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:10.258264 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:10.258214    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:10.258967 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:10.258948    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:08:10.706269 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:10.706186    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:10.706656 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:10.706605    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:10.708037 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:10.708015    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:08:30.714434 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:08:30.714351    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:08:41.763120 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.763085    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"]
Nov 03 02:08:41.767483 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.767463    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.781516 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.781484    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.781516 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.781518    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.781688 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.781604    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.781688 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.781656    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0ba8522-d62e-4f2e-a556-68599dd34e95-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.781688 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.781684    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z9ks\" (UniqueName: \"kubernetes.io/projected/d0ba8522-d62e-4f2e-a556-68599dd34e95-kube-api-access-2z9ks\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.781834 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.781787    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.781834 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.781828    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.781944 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.781851    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.782000 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.781968    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.816297 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.816258    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f7d9bb-openshift-default-dockercfg-dp5lm\""
Nov 03 02:08:41.867059 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.867025    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"]
Nov 03 02:08:41.883045 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.883012    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.883045 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.883057    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0ba8522-d62e-4f2e-a556-68599dd34e95-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.883312 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.883084    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z9ks\" (UniqueName: \"kubernetes.io/projected/d0ba8522-d62e-4f2e-a556-68599dd34e95-kube-api-access-2z9ks\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.883312 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.883287    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.883432 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.883350    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.883432 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.883403    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.883540 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.883450    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.883540 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.883499    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.883540 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.883525    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.883766 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.883737    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.884265 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.883499    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.884265 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.883929    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.884265 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.884064    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0ba8522-d62e-4f2e-a556-68599dd34e95-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.884469 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.884447    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.886294 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.886269    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.886670 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.886651    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.940267 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.940231    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:41.940539 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:41.940512    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z9ks\" (UniqueName: \"kubernetes.io/projected/d0ba8522-d62e-4f2e-a556-68599dd34e95-kube-api-access-2z9ks\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:42.082242 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:42.082161    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:42.508909 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:42.508878    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"]
Nov 03 02:08:42.514650 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:08:42.514621    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ba8522_d62e_4f2e_a556_68599dd34e95.slice/crio-8fe5695245a94d022cba33402cc2e7a6bd36b1a4a7a69e4d1477f76bb067b37e WatchSource:0}: Error finding container 8fe5695245a94d022cba33402cc2e7a6bd36b1a4a7a69e4d1477f76bb067b37e: Status 404 returned error can't find the container with id 8fe5695245a94d022cba33402cc2e7a6bd36b1a4a7a69e4d1477f76bb067b37e
Nov 03 02:08:42.516894 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:42.516860    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:08:42.516977 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:42.516929    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:08:42.516977 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:42.516961    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:08:43.208316 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:43.208200    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277" event={"ID":"d0ba8522-d62e-4f2e-a556-68599dd34e95","Type":"ContainerStarted","Data":"0e48605739173628a92ac95558f5c74152fb783f317144d8738b256f9e768202"}
Nov 03 02:08:43.208316 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:43.208275    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277" event={"ID":"d0ba8522-d62e-4f2e-a556-68599dd34e95","Type":"ContainerStarted","Data":"8fe5695245a94d022cba33402cc2e7a6bd36b1a4a7a69e4d1477f76bb067b37e"}
Nov 03 02:08:43.365090 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:43.365030    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277" podStartSLOduration=2.365015357 podStartE2EDuration="2.365015357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:08:43.363357534 +0000 UTC m=+1050.377598901" watchObservedRunningTime="2026-11-03 02:08:43.365015357 +0000 UTC m=+1050.379256912"
Nov 03 02:08:44.082844 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:44.082808    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:44.084681 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:44.084658    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 02:08:44.084762 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:44.084732    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277" podUID="d0ba8522-d62e-4f2e-a556-68599dd34e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 02:08:45.082775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:45.082731    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 02:08:45.083186 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:45.082798    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277" podUID="d0ba8522-d62e-4f2e-a556-68599dd34e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 02:08:46.082752 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:46.082714    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 02:08:46.082935 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:46.082769    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277" podUID="d0ba8522-d62e-4f2e-a556-68599dd34e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 02:08:47.083003 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:47.082964    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 02:08:47.083404 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:47.083030    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277" podUID="d0ba8522-d62e-4f2e-a556-68599dd34e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 02:08:48.082528 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:48.082490    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 02:08:48.082729 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:48.082544    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277" podUID="d0ba8522-d62e-4f2e-a556-68599dd34e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 02:08:49.083080 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:49.083043    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 02:08:49.083583 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:49.083096    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277" podUID="d0ba8522-d62e-4f2e-a556-68599dd34e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 02:08:50.087244 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:50.087212    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:50.087707 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:50.087671    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:08:50.088462 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:08:50.088443    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:09:00.713370 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:09:00.713329    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:09:30.713871 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:09:30.713833    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:09:32.792264 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.792228    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"]
Nov 03 02:09:32.796113 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.796091    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:32.875461 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.875427    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f526d9-2fd8-4c8d-9175-732afdbf3570-tls-certs\") pod \"distinct-2-w2-521f1f-kserve-896d88b95-bs4pm\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:32.875639 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.875482    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-dshm\") pod \"distinct-2-w2-521f1f-kserve-896d88b95-bs4pm\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:32.875731 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.875706    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-model-cache\") pod \"distinct-2-w2-521f1f-kserve-896d88b95-bs4pm\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:32.875799 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.875754    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-home\") pod \"distinct-2-w2-521f1f-kserve-896d88b95-bs4pm\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:32.875856 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.875821    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-tmp-dir\") pod \"distinct-2-w2-521f1f-kserve-896d88b95-bs4pm\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:32.875924 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.875895    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tmr2\" (UniqueName: \"kubernetes.io/projected/b1f526d9-2fd8-4c8d-9175-732afdbf3570-kube-api-access-4tmr2\") pod \"distinct-2-w2-521f1f-kserve-896d88b95-bs4pm\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:32.892791 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.892755    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-521f1f\"/\"default-dockercfg-dkd2g\""
Nov 03 02:09:32.892791 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.892755    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-521f1f\"/\"distinct-2-w2-521f1f-kserve-self-signed-certs\""
Nov 03 02:09:32.892791 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.892754    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-521f1f\"/\"openshift-service-ca.crt\""
Nov 03 02:09:32.893075 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.892928    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-521f1f\"/\"kube-root-ca.crt\""
Nov 03 02:09:32.896187 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.896156    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"]
Nov 03 02:09:32.977160 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.977128    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f526d9-2fd8-4c8d-9175-732afdbf3570-tls-certs\") pod \"distinct-2-w2-521f1f-kserve-896d88b95-bs4pm\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:32.977377 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.977172    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-dshm\") pod \"distinct-2-w2-521f1f-kserve-896d88b95-bs4pm\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:32.977377 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.977283    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-model-cache\") pod \"distinct-2-w2-521f1f-kserve-896d88b95-bs4pm\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:32.977377 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.977315    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-home\") pod \"distinct-2-w2-521f1f-kserve-896d88b95-bs4pm\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:32.977377 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.977370    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-tmp-dir\") pod \"distinct-2-w2-521f1f-kserve-896d88b95-bs4pm\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:32.977600 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.977397    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tmr2\" (UniqueName: \"kubernetes.io/projected/b1f526d9-2fd8-4c8d-9175-732afdbf3570-kube-api-access-4tmr2\") pod \"distinct-2-w2-521f1f-kserve-896d88b95-bs4pm\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:32.977802 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.977774    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-model-cache\") pod \"distinct-2-w2-521f1f-kserve-896d88b95-bs4pm\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:32.977902 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.977814    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-home\") pod \"distinct-2-w2-521f1f-kserve-896d88b95-bs4pm\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:32.977902 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.977861    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-tmp-dir\") pod \"distinct-2-w2-521f1f-kserve-896d88b95-bs4pm\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:32.979656 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.979630    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-dshm\") pod \"distinct-2-w2-521f1f-kserve-896d88b95-bs4pm\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:32.979905 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:32.979890    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f526d9-2fd8-4c8d-9175-732afdbf3570-tls-certs\") pod \"distinct-2-w2-521f1f-kserve-896d88b95-bs4pm\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:33.030836 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:33.030806    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tmr2\" (UniqueName: \"kubernetes.io/projected/b1f526d9-2fd8-4c8d-9175-732afdbf3570-kube-api-access-4tmr2\") pod \"distinct-2-w2-521f1f-kserve-896d88b95-bs4pm\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") " pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:33.111282 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:33.107237    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:33.482219 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:33.482142    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"]
Nov 03 02:09:33.536584 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:33.536554    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm" event={"ID":"b1f526d9-2fd8-4c8d-9175-732afdbf3570","Type":"ContainerStarted","Data":"cf3a1108cc5b9315e90bc3845bff75396bee009b9ebdeb9fa8a628b671058177"}
Nov 03 02:09:34.543412 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:34.543373    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm" event={"ID":"b1f526d9-2fd8-4c8d-9175-732afdbf3570","Type":"ContainerStarted","Data":"194e25547ee2ad1129d8f0cf1872e68d7729918aee728405d9adcea59477bdee"}
Nov 03 02:09:34.653163 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:34.653113    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm" podStartSLOduration=2.653098376 podStartE2EDuration="2.653098376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:09:34.648980273 +0000 UTC m=+1101.663221638" watchObservedRunningTime="2026-11-03 02:09:34.653098376 +0000 UTC m=+1101.667339743"
Nov 03 02:09:35.109200 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:35.109171    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:43.109386 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:43.109356    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:43.120968 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:43.120939    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:43.611994 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:43.611961    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:09:47.502857 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.502824    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"]
Nov 03 02:09:47.507195 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.507175    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.549477 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.549435    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"]
Nov 03 02:09:47.553541 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.553519    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.607194 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.607162    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"]
Nov 03 02:09:47.646577 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.646541    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.646577 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.646584    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4394660-3dbc-46ea-9dd4-e4d52123648d-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.646895 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.646606    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.646895 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.646712    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.646895 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.646797    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.646895 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.646819    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.647122 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.646927    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.647122 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.646982    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvhqp\" (UniqueName: \"kubernetes.io/projected/d4394660-3dbc-46ea-9dd4-e4d52123648d-kube-api-access-wvhqp\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.647122 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.647009    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.647122 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.647082    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.647122 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.647105    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbv7x\" (UniqueName: \"kubernetes.io/projected/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-kube-api-access-zbv7x\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.647395 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.647136    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.647395 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.647203    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.647395 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.647224    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.647395 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.647250    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.647395 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.647266    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.647395 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.647355    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.647395 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.647379    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.652874 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.652845    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"]
Nov 03 02:09:47.748705 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.748651    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.748887 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.748769    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvhqp\" (UniqueName: \"kubernetes.io/projected/d4394660-3dbc-46ea-9dd4-e4d52123648d-kube-api-access-wvhqp\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.748887 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.748811    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.748887 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.748859    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.749058 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.748890    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbv7x\" (UniqueName: \"kubernetes.io/projected/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-kube-api-access-zbv7x\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.749058 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.748938    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.749058 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.748976    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.749058 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.749007    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.749058 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.749049    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.749344 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.749074    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.749344 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.749100    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.749344 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.749112    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.749344 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.749169    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.749344 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.749260    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.749344 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.749297    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4394660-3dbc-46ea-9dd4-e4d52123648d-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.749344 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.749326    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.749344 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.749328    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.749814 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.749433    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.749814 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.749674    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.749929 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.749848    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.749989 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.749969    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.750156 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.750137    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.750390 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.750363    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4394660-3dbc-46ea-9dd4-e4d52123648d-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.750544 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.750528    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.750625 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.750567    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.751082 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.751041    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.751308 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.751262    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.751393 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.751314    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.752024 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.751995    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.752733 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.752039    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.752803 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.752769    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.753213 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.753170    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.803239 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.803195    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.803422 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.803265    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvhqp\" (UniqueName: \"kubernetes.io/projected/d4394660-3dbc-46ea-9dd4-e4d52123648d-kube-api-access-wvhqp\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.803422 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.803308    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbv7x\" (UniqueName: \"kubernetes.io/projected/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-kube-api-access-zbv7x\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.804859 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.804837    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:47.821271 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.821235    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:47.867013 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:47.866979    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:48.263460 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:48.262856    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"]
Nov 03 02:09:48.266668 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:48.266637    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:09:48.266783 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:48.266715    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:09:48.266783 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:48.266738    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"]
Nov 03 02:09:48.266783 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:48.266753    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:09:48.302017 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:09:48.301988    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e48d72_ce80_4a8e_ada5_c5f7f3cda02f.slice/crio-6773200885dfb4170a019544e8ce091f206dca5645c2b1e6a1f2a9a3e2259be7 WatchSource:0}: Error finding container 6773200885dfb4170a019544e8ce091f206dca5645c2b1e6a1f2a9a3e2259be7: Status 404 returned error can't find the container with id 6773200885dfb4170a019544e8ce091f206dca5645c2b1e6a1f2a9a3e2259be7
Nov 03 02:09:48.304415 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:48.304383    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:09:48.304495 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:48.304453    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:09:48.304495 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:48.304484    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:09:48.639114 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:48.639008    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk" event={"ID":"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f","Type":"ContainerStarted","Data":"5fe083e68b42ea772960b4ccb1eb98acc68d1a563bca20523e38594c7b967186"}
Nov 03 02:09:48.639114 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:48.639077    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk" event={"ID":"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f","Type":"ContainerStarted","Data":"6773200885dfb4170a019544e8ce091f206dca5645c2b1e6a1f2a9a3e2259be7"}
Nov 03 02:09:48.641057 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:48.641022    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z" event={"ID":"d4394660-3dbc-46ea-9dd4-e4d52123648d","Type":"ContainerStarted","Data":"c8736a3da247a0131f454e8f034a940fe9ab54fd38b089e48256bda3d3b36a5d"}
Nov 03 02:09:48.641176 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:48.641076    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z" event={"ID":"d4394660-3dbc-46ea-9dd4-e4d52123648d","Type":"ContainerStarted","Data":"c877cd5a7d13cca90bfd64fa7990f964e7c9338a46140c18c2f1e99ba7875b46"}
Nov 03 02:09:48.748926 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:48.748868    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk" podStartSLOduration=1.7488498209999999 podStartE2EDuration="1.748849821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:09:48.748499751 +0000 UTC m=+1115.762741119" watchObservedRunningTime="2026-11-03 02:09:48.748849821 +0000 UTC m=+1115.763091187"
Nov 03 02:09:48.821457 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:48.821421    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:48.859771 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:48.857983    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z" podStartSLOduration=1.857966426 podStartE2EDuration="1.857966426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:09:48.857067251 +0000 UTC m=+1115.871308628" watchObservedRunningTime="2026-11-03 02:09:48.857966426 +0000 UTC m=+1115.872207793"
Nov 03 02:09:48.867747 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:48.867714    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:49.831910 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:49.831868    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 02:09:49.832562 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:49.832529    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z" podUID="d4394660-3dbc-46ea-9dd4-e4d52123648d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 02:09:49.869156 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:49.869119    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 02:09:49.869394 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:49.869373    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk" podUID="28e48d72-ce80-4a8e-ada5-c5f7f3cda02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 02:09:50.824608 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:50.824135    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 02:09:50.824608 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:50.824205    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z" podUID="d4394660-3dbc-46ea-9dd4-e4d52123648d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 02:09:50.868433 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:50.868205    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 02:09:50.868433 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:50.868271    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk" podUID="28e48d72-ce80-4a8e-ada5-c5f7f3cda02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 02:09:51.825579 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:51.825119    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 02:09:51.825579 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:51.825185    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z" podUID="d4394660-3dbc-46ea-9dd4-e4d52123648d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 02:09:51.868344 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:51.868307    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 02:09:51.868524 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:51.868371    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk" podUID="28e48d72-ce80-4a8e-ada5-c5f7f3cda02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 02:09:52.822776 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:52.822128    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 02:09:52.822776 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:52.822196    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z" podUID="d4394660-3dbc-46ea-9dd4-e4d52123648d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 02:09:52.867386 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:52.867353    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 02:09:52.867566 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:52.867408    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk" podUID="28e48d72-ce80-4a8e-ada5-c5f7f3cda02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 02:09:53.823181 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:53.823131    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 02:09:53.823878 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:53.823847    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z" podUID="d4394660-3dbc-46ea-9dd4-e4d52123648d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 02:09:53.868098 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:53.868057    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 02:09:53.868289 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:53.868136    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk" podUID="28e48d72-ce80-4a8e-ada5-c5f7f3cda02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 02:09:54.822142 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:54.822099    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 03 02:09:54.822369 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:54.822167    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z" podUID="d4394660-3dbc-46ea-9dd4-e4d52123648d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 03 02:09:54.867947 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:54.867906    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 03 02:09:54.868497 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:54.867974    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk" podUID="28e48d72-ce80-4a8e-ada5-c5f7f3cda02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 03 02:09:55.939959 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:55.938547    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:55.939959 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:55.939474    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:55.941164 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:55.941140    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:09:56.057076 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:56.057044    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:56.057356 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:56.057332    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:09:56.058302 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:09:56.058281    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:10:00.714520 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:10:00.714478    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:10:15.195351 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.195312    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"]
Nov 03 02:10:15.201052 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.201026    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.286166 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.286122    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.286348 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.286200    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.286396 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.286350    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.286396 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.286387    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.286499 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.286463    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.286612 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.286573    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.286660 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.286648    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.286755 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.286739    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnzbv\" (UniqueName: \"kubernetes.io/projected/2ac8ea76-22ba-4392-a97c-ca67dc13f427-kube-api-access-hnzbv\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.286873 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.286854    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.298740 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.298679    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-521f1f-openshift-default-dockercfg-c9pnp\""
Nov 03 02:10:15.301367 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.301340    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"]
Nov 03 02:10:15.383680 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.383649    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-nqtz4-56vtt"]
Nov 03 02:10:15.387681 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.387664    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nqtz4-56vtt"
Nov 03 02:10:15.387781 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.387679    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.387781 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.387734    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnzbv\" (UniqueName: \"kubernetes.io/projected/2ac8ea76-22ba-4392-a97c-ca67dc13f427-kube-api-access-hnzbv\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.387850 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.387793    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.387850 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.387827    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.387923 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.387859    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.387994 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.387944    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.388049 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.388009    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.388106 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.388050    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.388106 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.388093    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.388240 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.388114    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.388310 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.388237    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.388537 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.388515    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.388671 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.388643    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.388796 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.388777    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.390827 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.390803    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.390993 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.390975    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.442550 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.442497    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnzbv\" (UniqueName: \"kubernetes.io/projected/2ac8ea76-22ba-4392-a97c-ca67dc13f427-kube-api-access-hnzbv\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.442770 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.442657    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.449223 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.449156    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-nqtz4-56vtt"]
Nov 03 02:10:15.489575 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.489536    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrbv9\" (UniqueName: \"kubernetes.io/projected/9ec85f2a-a422-4083-b5d1-d753aefdff63-kube-api-access-vrbv9\") pod \"periodic-gathering-nqtz4-56vtt\" (UID: \"9ec85f2a-a422-4083-b5d1-d753aefdff63\") " pod="openshift-insights/periodic-gathering-nqtz4-56vtt"
Nov 03 02:10:15.489777 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.489608    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ec85f2a-a422-4083-b5d1-d753aefdff63-trusted-ca-bundle\") pod \"periodic-gathering-nqtz4-56vtt\" (UID: \"9ec85f2a-a422-4083-b5d1-d753aefdff63\") " pod="openshift-insights/periodic-gathering-nqtz4-56vtt"
Nov 03 02:10:15.489777 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.489742    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ec85f2a-a422-4083-b5d1-d753aefdff63-service-ca-bundle\") pod \"periodic-gathering-nqtz4-56vtt\" (UID: \"9ec85f2a-a422-4083-b5d1-d753aefdff63\") " pod="openshift-insights/periodic-gathering-nqtz4-56vtt"
Nov 03 02:10:15.489863 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.489784    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9ec85f2a-a422-4083-b5d1-d753aefdff63-archives-path\") pod \"periodic-gathering-nqtz4-56vtt\" (UID: \"9ec85f2a-a422-4083-b5d1-d753aefdff63\") " pod="openshift-insights/periodic-gathering-nqtz4-56vtt"
Nov 03 02:10:15.515553 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.515517    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:15.591736 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.591655    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ec85f2a-a422-4083-b5d1-d753aefdff63-trusted-ca-bundle\") pod \"periodic-gathering-nqtz4-56vtt\" (UID: \"9ec85f2a-a422-4083-b5d1-d753aefdff63\") " pod="openshift-insights/periodic-gathering-nqtz4-56vtt"
Nov 03 02:10:15.591919 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.591788    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ec85f2a-a422-4083-b5d1-d753aefdff63-service-ca-bundle\") pod \"periodic-gathering-nqtz4-56vtt\" (UID: \"9ec85f2a-a422-4083-b5d1-d753aefdff63\") " pod="openshift-insights/periodic-gathering-nqtz4-56vtt"
Nov 03 02:10:15.591919 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.591840    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9ec85f2a-a422-4083-b5d1-d753aefdff63-archives-path\") pod \"periodic-gathering-nqtz4-56vtt\" (UID: \"9ec85f2a-a422-4083-b5d1-d753aefdff63\") " pod="openshift-insights/periodic-gathering-nqtz4-56vtt"
Nov 03 02:10:15.599174 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.591982    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrbv9\" (UniqueName: \"kubernetes.io/projected/9ec85f2a-a422-4083-b5d1-d753aefdff63-kube-api-access-vrbv9\") pod \"periodic-gathering-nqtz4-56vtt\" (UID: \"9ec85f2a-a422-4083-b5d1-d753aefdff63\") " pod="openshift-insights/periodic-gathering-nqtz4-56vtt"
Nov 03 02:10:15.599174 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.593422    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ec85f2a-a422-4083-b5d1-d753aefdff63-trusted-ca-bundle\") pod \"periodic-gathering-nqtz4-56vtt\" (UID: \"9ec85f2a-a422-4083-b5d1-d753aefdff63\") " pod="openshift-insights/periodic-gathering-nqtz4-56vtt"
Nov 03 02:10:15.599174 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.594084    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ec85f2a-a422-4083-b5d1-d753aefdff63-service-ca-bundle\") pod \"periodic-gathering-nqtz4-56vtt\" (UID: \"9ec85f2a-a422-4083-b5d1-d753aefdff63\") " pod="openshift-insights/periodic-gathering-nqtz4-56vtt"
Nov 03 02:10:15.599174 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.594372    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9ec85f2a-a422-4083-b5d1-d753aefdff63-archives-path\") pod \"periodic-gathering-nqtz4-56vtt\" (UID: \"9ec85f2a-a422-4083-b5d1-d753aefdff63\") " pod="openshift-insights/periodic-gathering-nqtz4-56vtt"
Nov 03 02:10:15.659632 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.659598    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrbv9\" (UniqueName: \"kubernetes.io/projected/9ec85f2a-a422-4083-b5d1-d753aefdff63-kube-api-access-vrbv9\") pod \"periodic-gathering-nqtz4-56vtt\" (UID: \"9ec85f2a-a422-4083-b5d1-d753aefdff63\") " pod="openshift-insights/periodic-gathering-nqtz4-56vtt"
Nov 03 02:10:15.715261 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.714789    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nqtz4-56vtt"
Nov 03 02:10:15.967778 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:15.965605    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"]
Nov 03 02:10:15.997443 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:10:15.997398    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac8ea76_22ba_4392_a97c_ca67dc13f427.slice/crio-d42dcea4f8856e28f772a560927b81f4610a17736504a5af88f805c208a9e771 WatchSource:0}: Error finding container d42dcea4f8856e28f772a560927b81f4610a17736504a5af88f805c208a9e771: Status 404 returned error can't find the container with id d42dcea4f8856e28f772a560927b81f4610a17736504a5af88f805c208a9e771
Nov 03 02:10:16.003420 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:16.002207    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:10:16.003420 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:16.002320    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:10:16.003420 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:16.002370    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:10:16.216416 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:10:16.216358    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec85f2a_a422_4083_b5d1_d753aefdff63.slice/crio-f1b7d7a220e0dc2b76f05457eb1fe4521578a9bb62a53330d4e5a6c21aa186c1 WatchSource:0}: Error finding container f1b7d7a220e0dc2b76f05457eb1fe4521578a9bb62a53330d4e5a6c21aa186c1: Status 404 returned error can't find the container with id f1b7d7a220e0dc2b76f05457eb1fe4521578a9bb62a53330d4e5a6c21aa186c1
Nov 03 02:10:16.221254 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:16.221132    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-nqtz4-56vtt"]
Nov 03 02:10:16.963481 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:16.963435    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nqtz4-56vtt" event={"ID":"9ec85f2a-a422-4083-b5d1-d753aefdff63","Type":"ContainerStarted","Data":"23d5d0a29b289dbe737abc1b2756694fc7df70e562763eca292fa5db0a570051"}
Nov 03 02:10:16.963891 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:16.963781    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nqtz4-56vtt" event={"ID":"9ec85f2a-a422-4083-b5d1-d753aefdff63","Type":"ContainerStarted","Data":"f1b7d7a220e0dc2b76f05457eb1fe4521578a9bb62a53330d4e5a6c21aa186c1"}
Nov 03 02:10:16.975717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:16.972488    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc" event={"ID":"2ac8ea76-22ba-4392-a97c-ca67dc13f427","Type":"ContainerStarted","Data":"c30c93a191219e82f9044010f380d06b76cfb642fed78a2e6fea9572594097b0"}
Nov 03 02:10:16.975970 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:16.975945    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc" event={"ID":"2ac8ea76-22ba-4392-a97c-ca67dc13f427","Type":"ContainerStarted","Data":"d42dcea4f8856e28f772a560927b81f4610a17736504a5af88f805c208a9e771"}
Nov 03 02:10:17.096727 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:17.094174    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-nqtz4-56vtt" podStartSLOduration=2.094152999 podStartE2EDuration="2.094152999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:10:17.093554468 +0000 UTC m=+1144.107795835" watchObservedRunningTime="2026-11-03 02:10:17.094152999 +0000 UTC m=+1144.108394365"
Nov 03 02:10:17.211719 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:17.211479    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc" podStartSLOduration=2.211460595 podStartE2EDuration="2.211460595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:10:17.208205439 +0000 UTC m=+1144.222446807" watchObservedRunningTime="2026-11-03 02:10:17.211460595 +0000 UTC m=+1144.225701963"
Nov 03 02:10:17.520600 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:17.520565    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:17.521323 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:17.521238    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 02:10:17.521477 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:17.521454    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc" podUID="2ac8ea76-22ba-4392-a97c-ca67dc13f427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 02:10:17.604846 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:17.604791    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm" podUID="b1f526d9-2fd8-4c8d-9175-732afdbf3570" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 02:10:17.650277 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:17.650229    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td" podUID="7f39ea17-c4d7-4de2-bb69-c7d3012c5f77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:10:18.516721 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:18.516130    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 02:10:18.516721 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:18.516201    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc" podUID="2ac8ea76-22ba-4392-a97c-ca67dc13f427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 02:10:19.519718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:19.519057    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 02:10:19.519718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:19.519121    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc" podUID="2ac8ea76-22ba-4392-a97c-ca67dc13f427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 02:10:20.517024 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:20.516985    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 02:10:20.517306 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:20.517277    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc" podUID="2ac8ea76-22ba-4392-a97c-ca67dc13f427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 02:10:21.520724 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:21.520365    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 02:10:21.520724 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:21.520427    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc" podUID="2ac8ea76-22ba-4392-a97c-ca67dc13f427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 02:10:22.517718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:22.517095    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 02:10:22.517718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:22.517158    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc" podUID="2ac8ea76-22ba-4392-a97c-ca67dc13f427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 02:10:23.516706 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:23.516659    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 03 02:10:23.517081 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:23.516746    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc" podUID="2ac8ea76-22ba-4392-a97c-ca67dc13f427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 03 02:10:24.521039 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:24.521005    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:24.521406 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:24.521291    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:24.522026 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:24.522001    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:10:30.713769 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:10:30.713629    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:10:31.692181 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.692147    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"]
Nov 03 02:10:31.696290 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.696270    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.796443 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.796409    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"]
Nov 03 02:10:31.796958 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.796927    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.797070 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.797043    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.797352 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.797323    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.797512 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.797424    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.797512 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.797485    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.797675 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.797516    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.797675 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.797586    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44sxb\" (UniqueName: \"kubernetes.io/projected/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-kube-api-access-44sxb\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.797675 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.797637    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.797675 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.797661    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.898933 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.898894    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44sxb\" (UniqueName: \"kubernetes.io/projected/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-kube-api-access-44sxb\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.899101 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.898965    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.899101 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.898991    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.899101 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.899021    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.899101 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.899070    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.899270 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.899151    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.899270 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.899235    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.899374 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.899275    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.899374 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.899303    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.899567 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.899538    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-data\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.899711 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.899594    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-workload-certs\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.899795 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.899769    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-credential-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.899898 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.899878    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-workload-socket\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.900019 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.899995    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istiod-ca-cert\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.901336 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.901312    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-envoy\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.901622 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.901604    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-podinfo\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.951164 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.951111    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-token\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:31.951445 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:31.951420    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44sxb\" (UniqueName: \"kubernetes.io/projected/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-kube-api-access-44sxb\") pod \"e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") " pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:32.009813 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:32.009787    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:32.435234 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:32.435190    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"]
Nov 03 02:10:32.440520 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:10:32.440492    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f80420e_9ed1_46ea_9fb1_c5489c72ed4c.slice/crio-8859fc5a4c518503c23b173c3aca554d22503b00a157a59c1e980406ea2f674b WatchSource:0}: Error finding container 8859fc5a4c518503c23b173c3aca554d22503b00a157a59c1e980406ea2f674b: Status 404 returned error can't find the container with id 8859fc5a4c518503c23b173c3aca554d22503b00a157a59c1e980406ea2f674b
Nov 03 02:10:32.442824 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:32.442792    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:10:32.442909 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:32.442859    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:10:32.442909 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:32.442886    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:10:33.113062 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:33.113027    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb" event={"ID":"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c","Type":"ContainerStarted","Data":"c2b6723120ec8371477eb2312a2dfdf61636388050d269485462419f95b360b5"}
Nov 03 02:10:33.113466 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:33.113090    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb" event={"ID":"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c","Type":"ContainerStarted","Data":"8859fc5a4c518503c23b173c3aca554d22503b00a157a59c1e980406ea2f674b"}
Nov 03 02:10:33.324608 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:33.324544    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb" podStartSLOduration=2.324523633 podStartE2EDuration="2.324523633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:10:33.273329984 +0000 UTC m=+1160.287571363" watchObservedRunningTime="2026-11-03 02:10:33.324523633 +0000 UTC m=+1160.338765001"
Nov 03 02:10:34.023007 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:34.022966    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:34.024819 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:34.024785    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 02:10:34.024957 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:34.024835    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb" podUID="6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 02:10:35.017729 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:35.017329    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 02:10:35.017729 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:35.017391    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb" podUID="6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 02:10:36.020717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:36.020080    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 02:10:36.020717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:36.020155    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb" podUID="6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 02:10:37.011717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:37.011106    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 02:10:37.011717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:37.011184    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb" podUID="6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 02:10:38.096717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:38.095810    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 02:10:38.096717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:38.095878    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb" podUID="6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 02:10:39.012557 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:39.012096    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 02:10:39.012557 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:39.012161    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb" podUID="6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 02:10:39.615823 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:39.615774    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td" podUID="7f39ea17-c4d7-4de2-bb69-c7d3012c5f77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:10:39.844882 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:39.844838    5005 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd" podUID="fd71fd74-9581-4c83-8954-55462e1d4438" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": context deadline exceeded"
Nov 03 02:10:40.109716 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:40.053345    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 02:10:40.109716 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:40.087766    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb" podUID="6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 02:10:42.051833 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:42.051798    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:10:42.052352 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:42.051944    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb" podUID="6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:10:42.257245 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:42.257216    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:42.257762 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:42.257729    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:42.258507 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:42.258489    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:10:56.302813 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:56.302772    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:10:56.303467 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:56.303433    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh" podUID="7d6ce282-686b-432c-bb78-7361aecf6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:10:56.598868 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:56.598773    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td" podUID="7f39ea17-c4d7-4de2-bb69-c7d3012c5f77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:10:56.800742 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:56.797998    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:10:56.800742 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:56.798076    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl" podUID="00145046-700c-436b-a729-4332d8935371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:10:57.031973 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:57.031936    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:10:57.032311 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:10:57.032277    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z" podUID="d4394660-3dbc-46ea-9dd4-e4d52123648d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:11:00.540188 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.540153    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"]
Nov 03 02:11:00.544576 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.544552    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.608420 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.608388    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.608610 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.608431    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.608682 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.608637    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.608757 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.608678    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwg8j\" (UniqueName: \"kubernetes.io/projected/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-kube-api-access-gwg8j\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.608804 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.608769    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.608804 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.608798    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.608889 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.608825    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.608889 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.608852    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.608964 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.608924    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.643380 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.643347    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"]
Nov 03 02:11:00.710286 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.710242    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.710662 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.710625    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwg8j\" (UniqueName: \"kubernetes.io/projected/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-kube-api-access-gwg8j\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.710965 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.710922    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.711119 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.711024    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.711119 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.711071    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.711260 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.711136    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.711260 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.711178    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.711419 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.711306    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.711615 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.711574    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.713024 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.712968    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.713376 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.713314    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.713734 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.713688    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.714370 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.714347    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.714475 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.714458    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.715146 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.715126    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.715890 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.715649    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.722285 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:11:00.722256    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:11:00.765666 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.765625    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.766201 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.766180    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwg8j\" (UniqueName: \"kubernetes.io/projected/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-kube-api-access-gwg8j\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:00.858780 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:00.858681    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:01.294362 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:01.293371    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"]
Nov 03 02:11:01.297634 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:11:01.297606    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e03236_c4ee_4370_9b6c_fcb997e3d4b5.slice/crio-336e08caa8623cd0b7574b657fa6079e8307b5c4bbabdeb1061a119d4a20a1b2 WatchSource:0}: Error finding container 336e08caa8623cd0b7574b657fa6079e8307b5c4bbabdeb1061a119d4a20a1b2: Status 404 returned error can't find the container with id 336e08caa8623cd0b7574b657fa6079e8307b5c4bbabdeb1061a119d4a20a1b2
Nov 03 02:11:01.299828 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:01.299784    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:11:01.299941 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:01.299859    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:11:01.299941 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:01.299889    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:11:01.770620 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:01.770580    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n" event={"ID":"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5","Type":"ContainerStarted","Data":"d8453d044a9c8507bfc59d07cc4502a20acb13b1ae5a1fdbe667e1e62517575d"}
Nov 03 02:11:01.771054 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:01.770652    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n" event={"ID":"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5","Type":"ContainerStarted","Data":"336e08caa8623cd0b7574b657fa6079e8307b5c4bbabdeb1061a119d4a20a1b2"}
Nov 03 02:11:01.858803 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:01.858769    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:01.880031 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:01.879971    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n" podStartSLOduration=1.8799534759999998 podStartE2EDuration="1.879953476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:11:01.878486088 +0000 UTC m=+1188.892727454" watchObservedRunningTime="2026-11-03 02:11:01.879953476 +0000 UTC m=+1188.894194843"
Nov 03 02:11:02.861239 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:02.861208    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 02:11:02.861654 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:02.861260    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n" podUID="a9e03236-c4ee-4370-9b6c-fcb997e3d4b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 02:11:03.859725 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:03.859671    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 02:11:03.859932 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:03.859747    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n" podUID="a9e03236-c4ee-4370-9b6c-fcb997e3d4b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 02:11:04.859727 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:04.859651    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 02:11:04.860206 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:04.859740    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n" podUID="a9e03236-c4ee-4370-9b6c-fcb997e3d4b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 02:11:05.859900 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:05.859809    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 02:11:05.859900 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:05.859875    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n" podUID="a9e03236-c4ee-4370-9b6c-fcb997e3d4b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 02:11:06.859740 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:06.859678    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 02:11:06.859922 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:06.859777    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n" podUID="a9e03236-c4ee-4370-9b6c-fcb997e3d4b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 02:11:07.859273 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:07.859234    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 02:11:07.859448 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:07.859302    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n" podUID="a9e03236-c4ee-4370-9b6c-fcb997e3d4b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 02:11:08.864876 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:08.864846    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:08.865360 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:08.865138    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:08.865746 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:08.865727    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 02:11:12.861266 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:12.861223    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"]
Nov 03 02:11:12.866864 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:12.866841    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:12.963849 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:12.963207    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:12.963849 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:12.963300    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf895979-b958-4260-8a57-9ca10a4501d7-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:12.963849 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:12.963330    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:12.963849 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:12.963368    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:12.963849 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:12.963391    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf895979-b958-4260-8a57-9ca10a4501d7-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:12.963849 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:12.963462    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:12.963849 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:12.963513    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv969\" (UniqueName: \"kubernetes.io/projected/bf895979-b958-4260-8a57-9ca10a4501d7-kube-api-access-hv969\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:12.963849 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:12.963608    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:12.963849 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:12.963643    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf895979-b958-4260-8a57-9ca10a4501d7-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:12.967397 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:12.967365    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"]
Nov 03 02:11:13.065823 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.065783    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf895979-b958-4260-8a57-9ca10a4501d7-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:13.066018 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.065891    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:13.066018 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.065974    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf895979-b958-4260-8a57-9ca10a4501d7-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:13.066018 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.065994    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:13.066210 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.066026    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:13.066210 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.066042    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf895979-b958-4260-8a57-9ca10a4501d7-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:13.066210 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.066102    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:13.066210 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.066151    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv969\" (UniqueName: \"kubernetes.io/projected/bf895979-b958-4260-8a57-9ca10a4501d7-kube-api-access-hv969\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:13.066412 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.066257    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:13.066470 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.066427    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:13.066591 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.066568    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:13.066745 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.066684    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:13.066990 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.066939    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:13.067131 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.067095    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf895979-b958-4260-8a57-9ca10a4501d7-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:13.069266 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.069241    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf895979-b958-4260-8a57-9ca10a4501d7-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:13.069934 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.069910    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:13.118741 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.118652    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf895979-b958-4260-8a57-9ca10a4501d7-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:13.120034 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.120015    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv969\" (UniqueName: \"kubernetes.io/projected/bf895979-b958-4260-8a57-9ca10a4501d7-kube-api-access-hv969\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:13.180419 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.180392    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:13.247446 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.247416    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:11:13.249714 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.249683    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:11:13.617656 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.617622    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"]
Nov 03 02:11:13.628313 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:11:13.628284    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf895979_b958_4260_8a57_9ca10a4501d7.slice/crio-d8c98ae08379e1d356fa29eeb16b842ca2cdb9c499abcc6eaf1e013ef997020f WatchSource:0}: Error finding container d8c98ae08379e1d356fa29eeb16b842ca2cdb9c499abcc6eaf1e013ef997020f: Status 404 returned error can't find the container with id d8c98ae08379e1d356fa29eeb16b842ca2cdb9c499abcc6eaf1e013ef997020f
Nov 03 02:11:13.630753 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.630685    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:11:13.630873 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.630807    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:11:13.630873 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.630853    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:11:13.873775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.873665    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk" event={"ID":"bf895979-b958-4260-8a57-9ca10a4501d7","Type":"ContainerStarted","Data":"d525383963c354e8cfca91500d10808330313992c3206ea2f0a46cb489c5b3bf"}
Nov 03 02:11:13.873775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.873750    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk" event={"ID":"bf895979-b958-4260-8a57-9ca10a4501d7","Type":"ContainerStarted","Data":"d8c98ae08379e1d356fa29eeb16b842ca2cdb9c499abcc6eaf1e013ef997020f"}
Nov 03 02:11:13.982631 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:13.982521    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk" podStartSLOduration=1.982502943 podStartE2EDuration="1.982502943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:11:13.98175627 +0000 UTC m=+1200.995997636" watchObservedRunningTime="2026-11-03 02:11:13.982502943 +0000 UTC m=+1200.996744310"
Nov 03 02:11:14.181380 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:14.181309    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:14.183033 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:14.183011    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 02:11:14.183089 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:14.183057    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk" podUID="bf895979-b958-4260-8a57-9ca10a4501d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 02:11:15.180956 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.180918    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 02:11:15.181336 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.180966    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk" podUID="bf895979-b958-4260-8a57-9ca10a4501d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 02:11:15.635394 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.635358    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"]
Nov 03 02:11:15.639875 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.639852    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.695094 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.695067    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"]
Nov 03 02:11:15.830496 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.830459    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55279a3b-4345-47ba-b065-b9ac01da611d-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.830660 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.830512    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55279a3b-4345-47ba-b065-b9ac01da611d-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.830660 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.830595    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.830660 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.830625    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55279a3b-4345-47ba-b065-b9ac01da611d-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.830660 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.830655    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.830835 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.830673    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7qhc\" (UniqueName: \"kubernetes.io/projected/55279a3b-4345-47ba-b065-b9ac01da611d-kube-api-access-d7qhc\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.830835 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.830779    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.830835 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.830812    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.830980 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.830839    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.932536 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.932454    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.932536 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.932493    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55279a3b-4345-47ba-b065-b9ac01da611d-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.932536 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.932527    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.932853 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.932545    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7qhc\" (UniqueName: \"kubernetes.io/projected/55279a3b-4345-47ba-b065-b9ac01da611d-kube-api-access-d7qhc\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.932853 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.932585    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.932853 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.932611    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.932853 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.932653    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.932853 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.932835    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55279a3b-4345-47ba-b065-b9ac01da611d-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.933099 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.932888    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55279a3b-4345-47ba-b065-b9ac01da611d-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.933099 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.932961    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-workload-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.933264 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.933244    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55279a3b-4345-47ba-b065-b9ac01da611d-istiod-ca-cert\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.933357 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.933333    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-workload-certs\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.933420 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.933371    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-istio-data\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.933420 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.933398    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-credential-socket\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.935087 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.935057    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-istio-envoy\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.935338 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.935322    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55279a3b-4345-47ba-b065-b9ac01da611d-istio-podinfo\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.985640 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.985611    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55279a3b-4345-47ba-b065-b9ac01da611d-istio-token\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:15.985794 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:15.985774    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7qhc\" (UniqueName: \"kubernetes.io/projected/55279a3b-4345-47ba-b065-b9ac01da611d-kube-api-access-d7qhc\") pod \"e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") " pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:16.181104 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:16.181061    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 02:11:16.181496 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:16.181121    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk" podUID="bf895979-b958-4260-8a57-9ca10a4501d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 02:11:16.254026 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:16.253991    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:16.723103 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:16.723023    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"]
Nov 03 02:11:16.727972 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:11:16.727940    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55279a3b_4345_47ba_b065_b9ac01da611d.slice/crio-ca400936629a4813505b68a5fe5cf411d435ab76f1c3fc5ac552795d38c8ea37 WatchSource:0}: Error finding container ca400936629a4813505b68a5fe5cf411d435ab76f1c3fc5ac552795d38c8ea37: Status 404 returned error can't find the container with id ca400936629a4813505b68a5fe5cf411d435ab76f1c3fc5ac552795d38c8ea37
Nov 03 02:11:16.730419 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:16.730384    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:11:16.730497 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:16.730450    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:11:16.730497 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:16.730477    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:11:16.896517 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:16.896484    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr" event={"ID":"55279a3b-4345-47ba-b065-b9ac01da611d","Type":"ContainerStarted","Data":"effcc350c6ea4e4e0a553135528abb3ff4dd7cea2ec7860158c9a9576bf09d96"}
Nov 03 02:11:16.896681 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:16.896537    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr" event={"ID":"55279a3b-4345-47ba-b065-b9ac01da611d","Type":"ContainerStarted","Data":"ca400936629a4813505b68a5fe5cf411d435ab76f1c3fc5ac552795d38c8ea37"}
Nov 03 02:11:17.181709 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:17.181666    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 02:11:17.182093 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:17.181749    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk" podUID="bf895979-b958-4260-8a57-9ca10a4501d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 02:11:17.254706 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:17.254671    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:17.256101 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:17.256075    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 02:11:17.256228 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:17.256126    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr" podUID="55279a3b-4345-47ba-b065-b9ac01da611d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 02:11:18.181718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:18.181661    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 02:11:18.182102 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:18.181732    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk" podUID="bf895979-b958-4260-8a57-9ca10a4501d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 02:11:18.255288 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:18.255254    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 02:11:18.255461 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:18.255304    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr" podUID="55279a3b-4345-47ba-b065-b9ac01da611d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 02:11:19.181301 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:19.181268    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 02:11:19.181473 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:19.181327    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk" podUID="bf895979-b958-4260-8a57-9ca10a4501d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 02:11:19.255304 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:19.255272    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 02:11:19.255792 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:19.255465    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr" podUID="55279a3b-4345-47ba-b065-b9ac01da611d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 02:11:20.181622 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:20.181589    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Nov 03 02:11:20.181810 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:20.181643    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk" podUID="bf895979-b958-4260-8a57-9ca10a4501d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Nov 03 02:11:20.272725 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:20.259305    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 02:11:20.272725 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:20.259373    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr" podUID="55279a3b-4345-47ba-b065-b9ac01da611d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 02:11:21.261664 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:21.259135    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 02:11:21.261664 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:21.259202    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr" podUID="55279a3b-4345-47ba-b065-b9ac01da611d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 02:11:21.895718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:21.881752    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:21.895718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:21.882858    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:21.895718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:21.884906    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:11:22.124717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:22.123862    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr" podStartSLOduration=7.123843139 podStartE2EDuration="7.123843139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:11:17.03185245 +0000 UTC m=+1204.046093817" watchObservedRunningTime="2026-11-03 02:11:22.123843139 +0000 UTC m=+1209.138084505"
Nov 03 02:11:22.213771 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:22.212791    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd" podUID="fd71fd74-9581-4c83-8954-55462e1d4438" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:11:22.276723 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:22.267077    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 02:11:22.276723 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:22.267132    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr" podUID="55279a3b-4345-47ba-b065-b9ac01da611d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 02:11:22.599718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:22.598829    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td" podUID="7f39ea17-c4d7-4de2-bb69-c7d3012c5f77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:11:22.624719 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:22.611451    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm" podUID="b1f526d9-2fd8-4c8d-9175-732afdbf3570" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:11:22.941557 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:22.920101    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr" podUID="96c17dd0-a778-4d4d-851f-551c152453ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:11:23.234137 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:23.233995    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd" podUID="fd71fd74-9581-4c83-8954-55462e1d4438" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:11:23.524035 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:23.524001    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 02:11:23.527736 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:23.524315    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr" podUID="55279a3b-4345-47ba-b065-b9ac01da611d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 02:11:23.552746 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:23.545401    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:11:23.602805 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:23.602763    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td" podUID="7f39ea17-c4d7-4de2-bb69-c7d3012c5f77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:11:23.607200 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:23.607172    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:11:23.694887 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:23.694846    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm" podUID="b1f526d9-2fd8-4c8d-9175-732afdbf3570" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:11:23.698717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:23.695413    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:11:23.785973 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:23.785878    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:11:23.789828 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:23.789771    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w" podUID="06335e60-e878-4ad3-9630-fabee17ef5d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:11:24.058777 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:23.928764    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr" podUID="96c17dd0-a778-4d4d-851f-551c152453ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:11:24.058777 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:24.054859    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 02:11:24.159832 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:24.159786    5005 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm" podUID="b1f526d9-2fd8-4c8d-9175-732afdbf3570" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:11:24.305729 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:24.275836    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd" podUID="fd71fd74-9581-4c83-8954-55462e1d4438" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded"
Nov 03 02:11:24.332656 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:24.293898    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 02:11:24.332656 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:24.327306    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr" podUID="55279a3b-4345-47ba-b065-b9ac01da611d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 02:11:24.618771 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:24.614847    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td" podUID="7f39ea17-c4d7-4de2-bb69-c7d3012c5f77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:11:24.695742 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:24.695677    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm" podUID="b1f526d9-2fd8-4c8d-9175-732afdbf3570" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:11:25.064447 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:25.064418    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr"
Nov 03 02:11:25.273403 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:25.273365    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 02:11:25.273643 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:25.273617    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr" podUID="55279a3b-4345-47ba-b065-b9ac01da611d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 02:11:25.313826 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:25.313783    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd" podUID="fd71fd74-9581-4c83-8954-55462e1d4438" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:11:25.331997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:25.331897    5005 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td" podUID="7f39ea17-c4d7-4de2-bb69-c7d3012c5f77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:11:25.631768 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:25.631641    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td" podUID="7f39ea17-c4d7-4de2-bb69-c7d3012c5f77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:11:25.747838 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:25.747785    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm" podUID="b1f526d9-2fd8-4c8d-9175-732afdbf3570" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:11:26.092960 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:26.092926    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 02:11:26.095719 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:26.095688    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 02:11:26.288313 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:26.288270    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 02:11:26.288575 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:26.288533    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr" podUID="55279a3b-4345-47ba-b065-b9ac01da611d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 02:11:26.720779 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:26.720162    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 02:11:27.254470 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:27.254434    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 03 02:11:27.254885 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:27.254494    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr" podUID="55279a3b-4345-47ba-b065-b9ac01da611d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 03 02:11:29.002123 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:29.002090    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:29.002647 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:29.002623    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:29.003579 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:29.003559    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:11:30.713293 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:11:30.713255    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:11:38.444856 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:38.444819    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst"]
Nov 03 02:11:38.449244 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:38.449221    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst"
Nov 03 02:11:38.502103 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:38.502070    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst"]
Nov 03 02:11:38.615278 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:38.615242    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-701ba30f\""
Nov 03 02:11:38.615453 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:38.615395    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-701ba30f-dockercfg-df59g\""
Nov 03 02:11:38.616486 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:38.616450    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e520cfcb-166e-469a-ba57-739da7f713da-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst\" (UID: \"e520cfcb-166e-469a-ba57-739da7f713da\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst"
Nov 03 02:11:38.616641 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:38.616496    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47xb2\" (UniqueName: \"kubernetes.io/projected/e520cfcb-166e-469a-ba57-739da7f713da-kube-api-access-47xb2\") pod \"payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst\" (UID: \"e520cfcb-166e-469a-ba57-739da7f713da\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst"
Nov 03 02:11:38.640659 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:38.640625    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd"]
Nov 03 02:11:38.646231 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:38.646205    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd"
Nov 03 02:11:38.670545 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:38.670514    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd"]
Nov 03 02:11:38.717672 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:38.717597    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncmt9\" (UniqueName: \"kubernetes.io/projected/4a3cc135-6421-48ed-83ab-3c9c8944d191-kube-api-access-ncmt9\") pod \"payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd\" (UID: \"4a3cc135-6421-48ed-83ab-3c9c8944d191\") " pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd"
Nov 03 02:11:38.717672 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:38.717644    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a3cc135-6421-48ed-83ab-3c9c8944d191-plugins-config-volume\") pod \"payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd\" (UID: \"4a3cc135-6421-48ed-83ab-3c9c8944d191\") " pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd"
Nov 03 02:11:38.717868 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:38.717741    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e520cfcb-166e-469a-ba57-739da7f713da-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst\" (UID: \"e520cfcb-166e-469a-ba57-739da7f713da\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst"
Nov 03 02:11:38.717868 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:38.717762    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47xb2\" (UniqueName: \"kubernetes.io/projected/e520cfcb-166e-469a-ba57-739da7f713da-kube-api-access-47xb2\") pod \"payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst\" (UID: \"e520cfcb-166e-469a-ba57-739da7f713da\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst"
Nov 03 02:11:38.718456 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:38.718434    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e520cfcb-166e-469a-ba57-739da7f713da-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst\" (UID: \"e520cfcb-166e-469a-ba57-739da7f713da\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst"
Nov 03 02:11:38.771067 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:38.771036    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47xb2\" (UniqueName: \"kubernetes.io/projected/e520cfcb-166e-469a-ba57-739da7f713da-kube-api-access-47xb2\") pod \"payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst\" (UID: \"e520cfcb-166e-469a-ba57-739da7f713da\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst"
Nov 03 02:11:38.819062 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:38.819027    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmt9\" (UniqueName: \"kubernetes.io/projected/4a3cc135-6421-48ed-83ab-3c9c8944d191-kube-api-access-ncmt9\") pod \"payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd\" (UID: \"4a3cc135-6421-48ed-83ab-3c9c8944d191\") " pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd"
Nov 03 02:11:38.819220 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:38.819098    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a3cc135-6421-48ed-83ab-3c9c8944d191-plugins-config-volume\") pod \"payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd\" (UID: \"4a3cc135-6421-48ed-83ab-3c9c8944d191\") " pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd"
Nov 03 02:11:38.819891 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:38.819871    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a3cc135-6421-48ed-83ab-3c9c8944d191-plugins-config-volume\") pod \"payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd\" (UID: \"4a3cc135-6421-48ed-83ab-3c9c8944d191\") " pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd"
Nov 03 02:11:38.873428 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:38.873397    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncmt9\" (UniqueName: \"kubernetes.io/projected/4a3cc135-6421-48ed-83ab-3c9c8944d191-kube-api-access-ncmt9\") pod \"payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd\" (UID: \"4a3cc135-6421-48ed-83ab-3c9c8944d191\") " pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd"
Nov 03 02:11:38.960938 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:38.960904    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd"
Nov 03 02:11:39.063817 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:39.063785    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst"
Nov 03 02:11:39.342166 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:39.342124    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd"]
Nov 03 02:11:39.347267 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:11:39.347239    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a3cc135_6421_48ed_83ab_3c9c8944d191.slice/crio-96b9d2bb341fd312d7e85b8c6315845067ddb2ff022bdb79b608b94c88db8fc8 WatchSource:0}: Error finding container 96b9d2bb341fd312d7e85b8c6315845067ddb2ff022bdb79b608b94c88db8fc8: Status 404 returned error can't find the container with id 96b9d2bb341fd312d7e85b8c6315845067ddb2ff022bdb79b608b94c88db8fc8
Nov 03 02:11:39.476016 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:39.475977    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst"]
Nov 03 02:11:39.481163 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:11:39.481136    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode520cfcb_166e_469a_ba57_739da7f713da.slice/crio-b9ad72cc0d83ed73e83f061e532ef868d6780623053bb96b7b26f56fd924c426 WatchSource:0}: Error finding container b9ad72cc0d83ed73e83f061e532ef868d6780623053bb96b7b26f56fd924c426: Status 404 returned error can't find the container with id b9ad72cc0d83ed73e83f061e532ef868d6780623053bb96b7b26f56fd924c426
Nov 03 02:11:40.317357 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:40.317311    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd" event={"ID":"4a3cc135-6421-48ed-83ab-3c9c8944d191","Type":"ContainerStarted","Data":"648e571f10a3caf36440ca8badc163b8c7ff05fd2c448882db73979f10e090b9"}
Nov 03 02:11:40.317673 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:40.317383    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd"
Nov 03 02:11:40.317673 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:40.317410    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd" event={"ID":"4a3cc135-6421-48ed-83ab-3c9c8944d191","Type":"ContainerStarted","Data":"96b9d2bb341fd312d7e85b8c6315845067ddb2ff022bdb79b608b94c88db8fc8"}
Nov 03 02:11:40.319304 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:40.319268    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst" event={"ID":"e520cfcb-166e-469a-ba57-739da7f713da","Type":"ContainerStarted","Data":"4d1fca288a73ed220f67ed6a78b67920934f421cee33badb58cfd8ec9be3c40b"}
Nov 03 02:11:40.319455 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:40.319331    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst" event={"ID":"e520cfcb-166e-469a-ba57-739da7f713da","Type":"ContainerStarted","Data":"b9ad72cc0d83ed73e83f061e532ef868d6780623053bb96b7b26f56fd924c426"}
Nov 03 02:11:40.319455 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:40.319364    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst"
Nov 03 02:11:40.427476 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:40.427417    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd" podStartSLOduration=2.4274034159999998 podStartE2EDuration="2.427403416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:11:40.423401787 +0000 UTC m=+1227.437643155" watchObservedRunningTime="2026-11-03 02:11:40.427403416 +0000 UTC m=+1227.441644782"
Nov 03 02:11:40.530864 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:40.530806    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst" podStartSLOduration=2.530789538 podStartE2EDuration="2.530789538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:11:40.528440528 +0000 UTC m=+1227.542681895" watchObservedRunningTime="2026-11-03 02:11:40.530789538 +0000 UTC m=+1227.545030915"
Nov 03 02:11:51.329205 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:51.329169    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd"
Nov 03 02:11:51.329678 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:51.329297    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst"
Nov 03 02:11:53.407822 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:53.407786    5005 generic.go:356] "Generic (PLEG): container finished" podID="9ec85f2a-a422-4083-b5d1-d753aefdff63" containerID="23d5d0a29b289dbe737abc1b2756694fc7df70e562763eca292fa5db0a570051" exitCode=0
Nov 03 02:11:53.408212 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:53.407860    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nqtz4-56vtt" event={"ID":"9ec85f2a-a422-4083-b5d1-d753aefdff63","Type":"ContainerDied","Data":"23d5d0a29b289dbe737abc1b2756694fc7df70e562763eca292fa5db0a570051"}
Nov 03 02:11:54.576522 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:54.576488    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nqtz4-56vtt"
Nov 03 02:11:54.721196 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:54.721101    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec85f2a-a422-4083-b5d1-d753aefdff63-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9ec85f2a-a422-4083-b5d1-d753aefdff63-archives-path\") pod \"9ec85f2a-a422-4083-b5d1-d753aefdff63\" (UID: \"9ec85f2a-a422-4083-b5d1-d753aefdff63\") "
Nov 03 02:11:54.721196 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:54.721190    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec85f2a-a422-4083-b5d1-d753aefdff63-kube-api-access-vrbv9\" (UniqueName: \"kubernetes.io/projected/9ec85f2a-a422-4083-b5d1-d753aefdff63-kube-api-access-vrbv9\") pod \"9ec85f2a-a422-4083-b5d1-d753aefdff63\" (UID: \"9ec85f2a-a422-4083-b5d1-d753aefdff63\") "
Nov 03 02:11:54.721396 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:54.721245    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec85f2a-a422-4083-b5d1-d753aefdff63-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ec85f2a-a422-4083-b5d1-d753aefdff63-service-ca-bundle\") pod \"9ec85f2a-a422-4083-b5d1-d753aefdff63\" (UID: \"9ec85f2a-a422-4083-b5d1-d753aefdff63\") "
Nov 03 02:11:54.721477 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:54.721459    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec85f2a-a422-4083-b5d1-d753aefdff63-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ec85f2a-a422-4083-b5d1-d753aefdff63-trusted-ca-bundle\") pod \"9ec85f2a-a422-4083-b5d1-d753aefdff63\" (UID: \"9ec85f2a-a422-4083-b5d1-d753aefdff63\") "
Nov 03 02:11:54.721549 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:54.721513    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec85f2a-a422-4083-b5d1-d753aefdff63-archives-path" pod "9ec85f2a-a422-4083-b5d1-d753aefdff63" (UID: "9ec85f2a-a422-4083-b5d1-d753aefdff63"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:11:54.721894 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:54.721858    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec85f2a-a422-4083-b5d1-d753aefdff63-service-ca-bundle" pod "9ec85f2a-a422-4083-b5d1-d753aefdff63" (UID: "9ec85f2a-a422-4083-b5d1-d753aefdff63"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:11:54.722030 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:54.721923    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec85f2a-a422-4083-b5d1-d753aefdff63-trusted-ca-bundle" pod "9ec85f2a-a422-4083-b5d1-d753aefdff63" (UID: "9ec85f2a-a422-4083-b5d1-d753aefdff63"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:11:54.722250 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:54.722222    5005 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ec85f2a-a422-4083-b5d1-d753aefdff63-trusted-ca-bundle\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:54.722250 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:54.722247    5005 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9ec85f2a-a422-4083-b5d1-d753aefdff63-archives-path\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:54.722346 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:54.722261    5005 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ec85f2a-a422-4083-b5d1-d753aefdff63-service-ca-bundle\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:54.724049 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:54.724022    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec85f2a-a422-4083-b5d1-d753aefdff63-kube-api-access-vrbv9" pod "9ec85f2a-a422-4083-b5d1-d753aefdff63" (UID: "9ec85f2a-a422-4083-b5d1-d753aefdff63"). InnerVolumeSpecName "kube-api-access-vrbv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:11:54.823221 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:54.823187    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrbv9\" (UniqueName: \"kubernetes.io/projected/9ec85f2a-a422-4083-b5d1-d753aefdff63-kube-api-access-vrbv9\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:55.421747 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:55.421645    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nqtz4-56vtt"
Nov 03 02:11:55.421747 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:55.421686    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nqtz4-56vtt" event={"ID":"9ec85f2a-a422-4083-b5d1-d753aefdff63","Type":"ContainerDied","Data":"f1b7d7a220e0dc2b76f05457eb1fe4521578a9bb62a53330d4e5a6c21aa186c1"}
Nov 03 02:11:55.421747 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:55.421740    5005 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1b7d7a220e0dc2b76f05457eb1fe4521578a9bb62a53330d4e5a6c21aa186c1"
Nov 03 02:11:58.575223 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.575176    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"]
Nov 03 02:11:58.578041 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.578014    5005 cpu_manager.go:414] "containerMap: removing container" podUID="9ec85f2a-a422-4083-b5d1-d753aefdff63" containerName="insights-gathering"
Nov 03 02:11:58.578216 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.578199    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec85f2a-a422-4083-b5d1-d753aefdff63" containerName="insights-gathering"
Nov 03 02:11:58.578555 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.578538    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec85f2a-a422-4083-b5d1-d753aefdff63" containerName="insights-gathering"
Nov 03 02:11:58.589814 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.589787    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:58.640975 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.640942    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 02:11:58.641408 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.641321    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2k54n\""
Nov 03 02:11:58.641588 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.641345    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 02:11:58.663601 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.663564    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c36dba16-afb6-4239-aef1-4c3dfa4fa43d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4rgnn\" (UID: \"c36dba16-afb6-4239-aef1-4c3dfa4fa43d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:58.663886 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.663864    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36dba16-afb6-4239-aef1-4c3dfa4fa43d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4rgnn\" (UID: \"c36dba16-afb6-4239-aef1-4c3dfa4fa43d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:58.664156 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.664138    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c36dba16-afb6-4239-aef1-4c3dfa4fa43d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4rgnn\" (UID: \"c36dba16-afb6-4239-aef1-4c3dfa4fa43d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:58.664295 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.664276    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c36dba16-afb6-4239-aef1-4c3dfa4fa43d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4rgnn\" (UID: \"c36dba16-afb6-4239-aef1-4c3dfa4fa43d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:58.664440 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.664424    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzvr4\" (UniqueName: \"kubernetes.io/projected/c36dba16-afb6-4239-aef1-4c3dfa4fa43d-kube-api-access-vzvr4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4rgnn\" (UID: \"c36dba16-afb6-4239-aef1-4c3dfa4fa43d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:58.664577 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.664562    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c36dba16-afb6-4239-aef1-4c3dfa4fa43d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4rgnn\" (UID: \"c36dba16-afb6-4239-aef1-4c3dfa4fa43d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:58.664688 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.664671    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c36dba16-afb6-4239-aef1-4c3dfa4fa43d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4rgnn\" (UID: \"c36dba16-afb6-4239-aef1-4c3dfa4fa43d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:58.726274 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.726238    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 02:11:58.775527 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.775484    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c36dba16-afb6-4239-aef1-4c3dfa4fa43d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4rgnn\" (UID: \"c36dba16-afb6-4239-aef1-4c3dfa4fa43d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:58.775851 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.775825    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzvr4\" (UniqueName: \"kubernetes.io/projected/c36dba16-afb6-4239-aef1-4c3dfa4fa43d-kube-api-access-vzvr4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4rgnn\" (UID: \"c36dba16-afb6-4239-aef1-4c3dfa4fa43d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:58.776388 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.776365    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c36dba16-afb6-4239-aef1-4c3dfa4fa43d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4rgnn\" (UID: \"c36dba16-afb6-4239-aef1-4c3dfa4fa43d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:58.776520 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.776503    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c36dba16-afb6-4239-aef1-4c3dfa4fa43d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4rgnn\" (UID: \"c36dba16-afb6-4239-aef1-4c3dfa4fa43d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:58.776719 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.776682    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c36dba16-afb6-4239-aef1-4c3dfa4fa43d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4rgnn\" (UID: \"c36dba16-afb6-4239-aef1-4c3dfa4fa43d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:58.776869 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.776853    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36dba16-afb6-4239-aef1-4c3dfa4fa43d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4rgnn\" (UID: \"c36dba16-afb6-4239-aef1-4c3dfa4fa43d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:58.777168 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.777147    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c36dba16-afb6-4239-aef1-4c3dfa4fa43d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4rgnn\" (UID: \"c36dba16-afb6-4239-aef1-4c3dfa4fa43d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:58.780491 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.780458    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c36dba16-afb6-4239-aef1-4c3dfa4fa43d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4rgnn\" (UID: \"c36dba16-afb6-4239-aef1-4c3dfa4fa43d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:58.789145 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.782657    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36dba16-afb6-4239-aef1-4c3dfa4fa43d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4rgnn\" (UID: \"c36dba16-afb6-4239-aef1-4c3dfa4fa43d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:58.789145 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.783039    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c36dba16-afb6-4239-aef1-4c3dfa4fa43d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4rgnn\" (UID: \"c36dba16-afb6-4239-aef1-4c3dfa4fa43d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:58.794202 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.794115    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"]
Nov 03 02:11:58.806177 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.805190    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c36dba16-afb6-4239-aef1-4c3dfa4fa43d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4rgnn\" (UID: \"c36dba16-afb6-4239-aef1-4c3dfa4fa43d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:58.810567 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.810453    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c36dba16-afb6-4239-aef1-4c3dfa4fa43d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4rgnn\" (UID: \"c36dba16-afb6-4239-aef1-4c3dfa4fa43d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:58.851770 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.850786    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzvr4\" (UniqueName: \"kubernetes.io/projected/c36dba16-afb6-4239-aef1-4c3dfa4fa43d-kube-api-access-vzvr4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4rgnn\" (UID: \"c36dba16-afb6-4239-aef1-4c3dfa4fa43d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:58.909343 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.909273    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c36dba16-afb6-4239-aef1-4c3dfa4fa43d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4rgnn\" (UID: \"c36dba16-afb6-4239-aef1-4c3dfa4fa43d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:58.916882 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:58.916753    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:11:59.548720 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:59.537898    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"]
Nov 03 02:11:59.587647 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:11:59.587599    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc36dba16_afb6_4239_aef1_4c3dfa4fa43d.slice/crio-f544e02021ee3edfd37455eda88e5f69a92870733f06f3ffd616ab1fb2267943 WatchSource:0}: Error finding container f544e02021ee3edfd37455eda88e5f69a92870733f06f3ffd616ab1fb2267943: Status 404 returned error can't find the container with id f544e02021ee3edfd37455eda88e5f69a92870733f06f3ffd616ab1fb2267943
Nov 03 02:11:59.591137 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:11:59.591113    5005 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 02:12:00.603781 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:00.603683    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn" event={"ID":"c36dba16-afb6-4239-aef1-4c3dfa4fa43d","Type":"ContainerStarted","Data":"f544e02021ee3edfd37455eda88e5f69a92870733f06f3ffd616ab1fb2267943"}
Nov 03 02:12:00.755253 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:00.755205    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-h4htz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:12:00.755567 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:00.755518    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz" podUID="af839e9e-267b-4c27-b8cc-4e4e591580e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:12:00.961219 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:12:00.961126    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:12:06.068415 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:06.068378    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"]
Nov 03 02:12:06.073211 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:06.073192    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"
Nov 03 02:12:06.122634 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:06.122602    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-701ba30f-serving-cert\""
Nov 03 02:12:06.154851 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:06.154808    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-metrics-certs\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd\" (UID: \"3ec530be-e064-476f-b3e3-6cafd2d3d1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"
Nov 03 02:12:06.155053 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:06.154865    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn7xf\" (UniqueName: \"kubernetes.io/projected/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-kube-api-access-qn7xf\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd\" (UID: \"3ec530be-e064-476f-b3e3-6cafd2d3d1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"
Nov 03 02:12:06.155053 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:06.154934    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-maas-api-tls\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd\" (UID: \"3ec530be-e064-476f-b3e3-6cafd2d3d1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"
Nov 03 02:12:06.155053 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:06.155003    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-openshift-service-ca\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd\" (UID: \"3ec530be-e064-476f-b3e3-6cafd2d3d1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"
Nov 03 02:12:06.173895 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:06.173857    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"]
Nov 03 02:12:06.256441 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:06.256396    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-maas-api-tls\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd\" (UID: \"3ec530be-e064-476f-b3e3-6cafd2d3d1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"
Nov 03 02:12:06.256645 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:06.256494    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-openshift-service-ca\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd\" (UID: \"3ec530be-e064-476f-b3e3-6cafd2d3d1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"
Nov 03 02:12:06.256645 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:12:06.256624    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-serving-cert: secret "maas-api-e2e-mt-701ba30f-serving-cert" not found
Nov 03 02:12:06.256794 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:12:06.256722    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-maas-api-tls podName:3ec530be-e064-476f-b3e3-6cafd2d3d1d4 nodeName:}" failed. No retries permitted until 2026-11-03 02:12:06.756689259 +0000 UTC m=+1253.770930609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-maas-api-tls") pod "maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd" (UID: "3ec530be-e064-476f-b3e3-6cafd2d3d1d4") : secret "maas-api-e2e-mt-701ba30f-serving-cert" not found
Nov 03 02:12:06.256893 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:06.256867    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-metrics-certs\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd\" (UID: \"3ec530be-e064-476f-b3e3-6cafd2d3d1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"
Nov 03 02:12:06.256957 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:06.256919    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7xf\" (UniqueName: \"kubernetes.io/projected/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-kube-api-access-qn7xf\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd\" (UID: \"3ec530be-e064-476f-b3e3-6cafd2d3d1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"
Nov 03 02:12:06.257411 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:06.257391    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-openshift-service-ca\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd\" (UID: \"3ec530be-e064-476f-b3e3-6cafd2d3d1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"
Nov 03 02:12:06.259422 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:06.259403    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-metrics-certs\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd\" (UID: \"3ec530be-e064-476f-b3e3-6cafd2d3d1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"
Nov 03 02:12:06.318279 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:06.318241    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn7xf\" (UniqueName: \"kubernetes.io/projected/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-kube-api-access-qn7xf\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd\" (UID: \"3ec530be-e064-476f-b3e3-6cafd2d3d1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"
Nov 03 02:12:06.764002 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:06.763962    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-maas-api-tls\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd\" (UID: \"3ec530be-e064-476f-b3e3-6cafd2d3d1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"
Nov 03 02:12:06.766342 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:06.766308    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-maas-api-tls\") pod \"maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd\" (UID: \"3ec530be-e064-476f-b3e3-6cafd2d3d1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"
Nov 03 02:12:06.988461 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:06.988429    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"
Nov 03 02:12:07.366488 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:07.366403    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"]
Nov 03 02:12:07.372177 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:12:07.372141    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec530be_e064_476f_b3e3_6cafd2d3d1d4.slice/crio-3b8ab9747dc0360e0c03374abe2e65e9824d7feb5ecef57dc582a49fc975e8bb WatchSource:0}: Error finding container 3b8ab9747dc0360e0c03374abe2e65e9824d7feb5ecef57dc582a49fc975e8bb: Status 404 returned error can't find the container with id 3b8ab9747dc0360e0c03374abe2e65e9824d7feb5ecef57dc582a49fc975e8bb
Nov 03 02:12:07.655347 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:07.655252    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd" event={"ID":"3ec530be-e064-476f-b3e3-6cafd2d3d1d4","Type":"ContainerStarted","Data":"3b8ab9747dc0360e0c03374abe2e65e9824d7feb5ecef57dc582a49fc975e8bb"}
Nov 03 02:12:08.115690 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:08.115656    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:12:08.115801 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:08.115740    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:12:08.662840 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:08.662806    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn" event={"ID":"c36dba16-afb6-4239-aef1-4c3dfa4fa43d","Type":"ContainerStarted","Data":"ab1596820a7280c7480a21c0bd64862cec05c8b3605a45028e36d9f5ca387ed3"}
Nov 03 02:12:08.663294 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:08.662868    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:12:08.664527 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:08.664497    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd" event={"ID":"3ec530be-e064-476f-b3e3-6cafd2d3d1d4","Type":"ContainerStarted","Data":"bfedc3f61fa0d7606e62299fa8a6841887721c9644003a556ead72740069e666"}
Nov 03 02:12:08.664656 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:08.664585    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"
Nov 03 02:12:08.866315 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:08.866250    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn" podStartSLOduration=2.342954095 podStartE2EDuration="10.866235751s" totalImagesPullingTime="8.523281656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:11:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:12:08.817284104 +0000 UTC m=+1255.831525470" watchObservedRunningTime="2026-11-03 02:12:08.866235751 +0000 UTC m=+1255.880477115"
Nov 03 02:12:08.926441 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:08.926387    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd" podStartSLOduration=1.825271199 podStartE2EDuration="2.926372652s" totalImagesPullingTime="1.101101453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:12:08.924834776 +0000 UTC m=+1255.939076152" watchObservedRunningTime="2026-11-03 02:12:08.926372652 +0000 UTC m=+1255.940614017"
Nov 03 02:12:09.673064 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:09.673031    5005 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-4rgnn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 02:12:09.673449 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:09.673093    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn" podUID="c36dba16-afb6-4239-aef1-4c3dfa4fa43d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 02:12:12.671654 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:12.671620    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4rgnn"
Nov 03 02:12:13.332565 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.332530    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"]
Nov 03 02:12:13.338150 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.338127    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.347554 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.347519    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sccp7\" (UniqueName: \"kubernetes.io/projected/38604d86-a2f2-4b36-b644-64d7ed7444b0-kube-api-access-sccp7\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.347666 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.347646    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.347732 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.347686    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.347797 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.347777    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.347849 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.347808    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.348014 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.347990    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.348056 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.348041    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.348134 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.348119    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.348230 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.348215    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38604d86-a2f2-4b36-b644-64d7ed7444b0-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.418101 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.418065    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"]
Nov 03 02:12:13.449505 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.449472    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.449667 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.449637    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38604d86-a2f2-4b36-b644-64d7ed7444b0-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.449793 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.449729    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sccp7\" (UniqueName: \"kubernetes.io/projected/38604d86-a2f2-4b36-b644-64d7ed7444b0-kube-api-access-sccp7\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.449859 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.449799    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.449859 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.449832    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.449965 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.449881    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.450021 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.449989    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.450233 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.450205    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.450414 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.450390    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.450517 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.450494    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38604d86-a2f2-4b36-b644-64d7ed7444b0-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.450587 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.450522    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.450587 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.450560    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.450837 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.450819    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.450937 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.450904    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.452172 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.452147    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.452271 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.452254    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.502999 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.502957    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.503184 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.503031    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sccp7\" (UniqueName: \"kubernetes.io/projected/38604d86-a2f2-4b36-b644-64d7ed7444b0-kube-api-access-sccp7\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:13.652761 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:13.652638    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:14.009362 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:14.009327    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"]
Nov 03 02:12:14.129491 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:14.129451    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"]
Nov 03 02:12:14.136086 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:12:14.136050    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38604d86_a2f2_4b36_b644_64d7ed7444b0.slice/crio-d470995b427bce80e757ee8f7a12c4c9debfb38d21c58f3106fab5f56058d389 WatchSource:0}: Error finding container d470995b427bce80e757ee8f7a12c4c9debfb38d21c58f3106fab5f56058d389: Status 404 returned error can't find the container with id d470995b427bce80e757ee8f7a12c4c9debfb38d21c58f3106fab5f56058d389
Nov 03 02:12:14.142728 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:14.142676    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:12:14.142809 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:14.142762    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:12:14.142809 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:14.142792    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:12:14.689761 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:14.687827    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"
Nov 03 02:12:14.712764 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:14.709463    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh" event={"ID":"38604d86-a2f2-4b36-b644-64d7ed7444b0","Type":"ContainerStarted","Data":"0b016549b9272958d4581ce4521cdd11c36de1253d306e13eab9a3a4a6e40681"}
Nov 03 02:12:14.712764 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:14.709535    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh" event={"ID":"38604d86-a2f2-4b36-b644-64d7ed7444b0","Type":"ContainerStarted","Data":"d470995b427bce80e757ee8f7a12c4c9debfb38d21c58f3106fab5f56058d389"}
Nov 03 02:12:14.945780 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:14.929489    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh" podStartSLOduration=1.929468896 podStartE2EDuration="1.929468896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:12:14.920987624 +0000 UTC m=+1261.935229004" watchObservedRunningTime="2026-11-03 02:12:14.929468896 +0000 UTC m=+1261.943710262"
Nov 03 02:12:15.653203 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:15.653163    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:15.654831 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:15.654800    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 02:12:15.654986 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:15.654874    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh" podUID="38604d86-a2f2-4b36-b644-64d7ed7444b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 02:12:16.653342 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:16.653308    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 02:12:16.653754 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:16.653365    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh" podUID="38604d86-a2f2-4b36-b644-64d7ed7444b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 02:12:17.653211 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:17.653173    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 02:12:17.653380 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:17.653239    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh" podUID="38604d86-a2f2-4b36-b644-64d7ed7444b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 02:12:18.653607 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:18.653567    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 02:12:18.654043 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:18.653625    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh" podUID="38604d86-a2f2-4b36-b644-64d7ed7444b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 02:12:19.020859 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.020808    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w" podUID="7e33c474-783f-4077-9667-d405119e45f2" containerName="payload-processing" containerID="cri-o://18cc856fd1be14e4b6d5ec8bdd16ecff317da69308c5300ddda6e3dde831e411" gracePeriod=25
Nov 03 02:12:19.372456 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.372427    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"
Nov 03 02:12:19.474604 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.474564    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e33c474-783f-4077-9667-d405119e45f2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e33c474-783f-4077-9667-d405119e45f2-routing-overlay-volume\") pod \"7e33c474-783f-4077-9667-d405119e45f2\" (UID: \"7e33c474-783f-4077-9667-d405119e45f2\") "
Nov 03 02:12:19.474794 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.474715    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e33c474-783f-4077-9667-d405119e45f2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e33c474-783f-4077-9667-d405119e45f2-provider-credentials\") pod \"7e33c474-783f-4077-9667-d405119e45f2\" (UID: \"7e33c474-783f-4077-9667-d405119e45f2\") "
Nov 03 02:12:19.474794 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.474782    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e33c474-783f-4077-9667-d405119e45f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e33c474-783f-4077-9667-d405119e45f2-plugins-config-volume\") pod \"7e33c474-783f-4077-9667-d405119e45f2\" (UID: \"7e33c474-783f-4077-9667-d405119e45f2\") "
Nov 03 02:12:19.475128 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.475093    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e33c474-783f-4077-9667-d405119e45f2-routing-overlay-volume" pod "7e33c474-783f-4077-9667-d405119e45f2" (UID: "7e33c474-783f-4077-9667-d405119e45f2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:12:19.475263 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.475237    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e33c474-783f-4077-9667-d405119e45f2-plugins-config-volume" pod "7e33c474-783f-4077-9667-d405119e45f2" (UID: "7e33c474-783f-4077-9667-d405119e45f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:12:19.475504 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.475484    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e33c474-783f-4077-9667-d405119e45f2-kube-api-access-ghqrs\" (UniqueName: \"kubernetes.io/projected/7e33c474-783f-4077-9667-d405119e45f2-kube-api-access-ghqrs\") pod \"7e33c474-783f-4077-9667-d405119e45f2\" (UID: \"7e33c474-783f-4077-9667-d405119e45f2\") "
Nov 03 02:12:19.476121 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.476091    5005 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e33c474-783f-4077-9667-d405119e45f2-plugins-config-volume\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:12:19.476230 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.476123    5005 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e33c474-783f-4077-9667-d405119e45f2-routing-overlay-volume\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:12:19.477769 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.477737    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e33c474-783f-4077-9667-d405119e45f2-provider-credentials" pod "7e33c474-783f-4077-9667-d405119e45f2" (UID: "7e33c474-783f-4077-9667-d405119e45f2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:12:19.478014 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.477989    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e33c474-783f-4077-9667-d405119e45f2-kube-api-access-ghqrs" pod "7e33c474-783f-4077-9667-d405119e45f2" (UID: "7e33c474-783f-4077-9667-d405119e45f2"). InnerVolumeSpecName "kube-api-access-ghqrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:12:19.577932 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.577898    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghqrs\" (UniqueName: \"kubernetes.io/projected/7e33c474-783f-4077-9667-d405119e45f2-kube-api-access-ghqrs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:12:19.577932 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.577926    5005 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e33c474-783f-4077-9667-d405119e45f2-provider-credentials\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:12:19.653467 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.653377    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 02:12:19.653467 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.653432    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh" podUID="38604d86-a2f2-4b36-b644-64d7ed7444b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 02:12:19.757213 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.757183    5005 generic.go:356] "Generic (PLEG): container finished" podID="7e33c474-783f-4077-9667-d405119e45f2" containerID="18cc856fd1be14e4b6d5ec8bdd16ecff317da69308c5300ddda6e3dde831e411" exitCode=0
Nov 03 02:12:19.757680 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.757232    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"
Nov 03 02:12:19.757680 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.757268    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w" event={"ID":"7e33c474-783f-4077-9667-d405119e45f2","Type":"ContainerDied","Data":"18cc856fd1be14e4b6d5ec8bdd16ecff317da69308c5300ddda6e3dde831e411"}
Nov 03 02:12:19.757680 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.757333    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-664f597bf6-98v2w" event={"ID":"7e33c474-783f-4077-9667-d405119e45f2","Type":"ContainerDied","Data":"ef996ae0c6721774d8e069094cfe0c1b08536488301f76d60161e32f2c5a0f6d"}
Nov 03 02:12:19.757680 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.757367    5005 scope.go:122] "RemoveContainer" containerID="18cc856fd1be14e4b6d5ec8bdd16ecff317da69308c5300ddda6e3dde831e411"
Nov 03 02:12:19.768681 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.768661    5005 scope.go:122] "RemoveContainer" containerID="18cc856fd1be14e4b6d5ec8bdd16ecff317da69308c5300ddda6e3dde831e411"
Nov 03 02:12:19.768963 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:12:19.768944    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18cc856fd1be14e4b6d5ec8bdd16ecff317da69308c5300ddda6e3dde831e411\": container with ID starting with 18cc856fd1be14e4b6d5ec8bdd16ecff317da69308c5300ddda6e3dde831e411 not found: ID does not exist" containerID="18cc856fd1be14e4b6d5ec8bdd16ecff317da69308c5300ddda6e3dde831e411"
Nov 03 02:12:19.769030 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.768970    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18cc856fd1be14e4b6d5ec8bdd16ecff317da69308c5300ddda6e3dde831e411"} err="failed to get container status \"18cc856fd1be14e4b6d5ec8bdd16ecff317da69308c5300ddda6e3dde831e411\": rpc error: code = NotFound desc = could not find container \"18cc856fd1be14e4b6d5ec8bdd16ecff317da69308c5300ddda6e3dde831e411\": container with ID starting with 18cc856fd1be14e4b6d5ec8bdd16ecff317da69308c5300ddda6e3dde831e411 not found: ID does not exist"
Nov 03 02:12:19.927590 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.927492    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"]
Nov 03 02:12:19.931086 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:19.931063    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-664f597bf6-98v2w"]
Nov 03 02:12:20.653230 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:20.653192    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 02:12:20.653402 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:20.653263    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh" podUID="38604d86-a2f2-4b36-b644-64d7ed7444b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 02:12:21.112598 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:21.112558    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e33c474-783f-4077-9667-d405119e45f2" path="/var/lib/kubelet/pods/7e33c474-783f-4077-9667-d405119e45f2/volumes"
Nov 03 02:12:21.657923 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:21.657891    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:21.658315 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:21.658296    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:21.658962 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:12:21.658943    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:12:30.721880 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:12:30.721843    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:13:00.719114 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:13:00.719031    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:13:30.716802 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:13:30.716757    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:13:32.762763 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:32.762715    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"]
Nov 03 02:13:32.763400 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:32.763384    5005 cpu_manager.go:414] "containerMap: removing container" podUID="7e33c474-783f-4077-9667-d405119e45f2" containerName="payload-processing"
Nov 03 02:13:32.763452 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:32.763402    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e33c474-783f-4077-9667-d405119e45f2" containerName="payload-processing"
Nov 03 02:13:32.763522 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:32.763513    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e33c474-783f-4077-9667-d405119e45f2" containerName="payload-processing"
Nov 03 02:13:32.767098 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:32.767077    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"
Nov 03 02:13:32.836383 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:32.836345    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e0635a-54b1-497e-9e2c-66f6961eee54-openshift-service-ca\") pod \"maas-api-e2e-api-a-7f332d-649fcfd989-24plt\" (UID: \"50e0635a-54b1-497e-9e2c-66f6961eee54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"
Nov 03 02:13:32.836559 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:32.836482    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e0635a-54b1-497e-9e2c-66f6961eee54-metrics-certs\") pod \"maas-api-e2e-api-a-7f332d-649fcfd989-24plt\" (UID: \"50e0635a-54b1-497e-9e2c-66f6961eee54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"
Nov 03 02:13:32.836616 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:32.836585    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgs6v\" (UniqueName: \"kubernetes.io/projected/50e0635a-54b1-497e-9e2c-66f6961eee54-kube-api-access-fgs6v\") pod \"maas-api-e2e-api-a-7f332d-649fcfd989-24plt\" (UID: \"50e0635a-54b1-497e-9e2c-66f6961eee54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"
Nov 03 02:13:32.836807 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:32.836787    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50e0635a-54b1-497e-9e2c-66f6961eee54-maas-api-tls\") pod \"maas-api-e2e-api-a-7f332d-649fcfd989-24plt\" (UID: \"50e0635a-54b1-497e-9e2c-66f6961eee54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"
Nov 03 02:13:32.851885 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:32.851841    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-7f332d-serving-cert\""
Nov 03 02:13:32.867166 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:32.867126    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"]
Nov 03 02:13:32.938416 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:32.938376    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e0635a-54b1-497e-9e2c-66f6961eee54-openshift-service-ca\") pod \"maas-api-e2e-api-a-7f332d-649fcfd989-24plt\" (UID: \"50e0635a-54b1-497e-9e2c-66f6961eee54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"
Nov 03 02:13:32.938610 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:32.938444    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e0635a-54b1-497e-9e2c-66f6961eee54-metrics-certs\") pod \"maas-api-e2e-api-a-7f332d-649fcfd989-24plt\" (UID: \"50e0635a-54b1-497e-9e2c-66f6961eee54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"
Nov 03 02:13:32.938610 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:32.938482    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgs6v\" (UniqueName: \"kubernetes.io/projected/50e0635a-54b1-497e-9e2c-66f6961eee54-kube-api-access-fgs6v\") pod \"maas-api-e2e-api-a-7f332d-649fcfd989-24plt\" (UID: \"50e0635a-54b1-497e-9e2c-66f6961eee54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"
Nov 03 02:13:32.938610 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:32.938568    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50e0635a-54b1-497e-9e2c-66f6961eee54-maas-api-tls\") pod \"maas-api-e2e-api-a-7f332d-649fcfd989-24plt\" (UID: \"50e0635a-54b1-497e-9e2c-66f6961eee54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"
Nov 03 02:13:32.938825 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:13:32.938760    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-serving-cert: secret "maas-api-e2e-api-a-7f332d-serving-cert" not found
Nov 03 02:13:32.938887 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:13:32.938858    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50e0635a-54b1-497e-9e2c-66f6961eee54-maas-api-tls podName:50e0635a-54b1-497e-9e2c-66f6961eee54 nodeName:}" failed. No retries permitted until 2026-11-03 02:13:33.438834275 +0000 UTC m=+1340.453075646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/50e0635a-54b1-497e-9e2c-66f6961eee54-maas-api-tls") pod "maas-api-e2e-api-a-7f332d-649fcfd989-24plt" (UID: "50e0635a-54b1-497e-9e2c-66f6961eee54") : secret "maas-api-e2e-api-a-7f332d-serving-cert" not found
Nov 03 02:13:32.939287 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:32.939264    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e0635a-54b1-497e-9e2c-66f6961eee54-openshift-service-ca\") pod \"maas-api-e2e-api-a-7f332d-649fcfd989-24plt\" (UID: \"50e0635a-54b1-497e-9e2c-66f6961eee54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"
Nov 03 02:13:32.941099 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:32.941079    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e0635a-54b1-497e-9e2c-66f6961eee54-metrics-certs\") pod \"maas-api-e2e-api-a-7f332d-649fcfd989-24plt\" (UID: \"50e0635a-54b1-497e-9e2c-66f6961eee54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"
Nov 03 02:13:32.992940 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:32.992904    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgs6v\" (UniqueName: \"kubernetes.io/projected/50e0635a-54b1-497e-9e2c-66f6961eee54-kube-api-access-fgs6v\") pod \"maas-api-e2e-api-a-7f332d-649fcfd989-24plt\" (UID: \"50e0635a-54b1-497e-9e2c-66f6961eee54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"
Nov 03 02:13:33.446013 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:33.445968    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50e0635a-54b1-497e-9e2c-66f6961eee54-maas-api-tls\") pod \"maas-api-e2e-api-a-7f332d-649fcfd989-24plt\" (UID: \"50e0635a-54b1-497e-9e2c-66f6961eee54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"
Nov 03 02:13:33.448537 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:33.448508    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50e0635a-54b1-497e-9e2c-66f6961eee54-maas-api-tls\") pod \"maas-api-e2e-api-a-7f332d-649fcfd989-24plt\" (UID: \"50e0635a-54b1-497e-9e2c-66f6961eee54\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"
Nov 03 02:13:33.681414 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:33.681379    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"
Nov 03 02:13:34.066969 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:34.066930    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"]
Nov 03 02:13:34.072853 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:13:34.072825    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e0635a_54b1_497e_9e2c_66f6961eee54.slice/crio-9fa6085572f856823a5d8bded4463baa8b819b23c9ac9fafb01e36f33e557e2d WatchSource:0}: Error finding container 9fa6085572f856823a5d8bded4463baa8b819b23c9ac9fafb01e36f33e557e2d: Status 404 returned error can't find the container with id 9fa6085572f856823a5d8bded4463baa8b819b23c9ac9fafb01e36f33e557e2d
Nov 03 02:13:34.378155 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:34.378043    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt" event={"ID":"50e0635a-54b1-497e-9e2c-66f6961eee54","Type":"ContainerStarted","Data":"9fa6085572f856823a5d8bded4463baa8b819b23c9ac9fafb01e36f33e557e2d"}
Nov 03 02:13:35.386132 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:35.386094    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt" event={"ID":"50e0635a-54b1-497e-9e2c-66f6961eee54","Type":"ContainerStarted","Data":"3741efb97dad30399eb4d28229b7f2d022eb18c1c74bea83ffd216321577d308"}
Nov 03 02:13:35.386616 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:35.386223    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"
Nov 03 02:13:35.496972 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:35.496918    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt" podStartSLOduration=2.5734527529999998 podStartE2EDuration="3.496902339s" totalImagesPullingTime="923.449586ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:13:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:13:35.492074679 +0000 UTC m=+1342.506316044" watchObservedRunningTime="2026-11-03 02:13:35.496902339 +0000 UTC m=+1342.511143706"
Nov 03 02:13:41.398775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:13:41.398742    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"
Nov 03 02:14:00.715592 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:14:00.715548    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:14:16.462155 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:16.462117    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"]
Nov 03 02:14:16.466547 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:16.466524    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"
Nov 03 02:14:16.516419 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:16.516385    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a772c3-serving-cert\""
Nov 03 02:14:16.522364 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:16.522331    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"]
Nov 03 02:14:16.589752 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:16.589665    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4637951-2a4e-4796-8b07-179acbdc0cfb-metrics-certs\") pod \"maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm\" (UID: \"c4637951-2a4e-4796-8b07-179acbdc0cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"
Nov 03 02:14:16.589942 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:16.589882    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmdcr\" (UniqueName: \"kubernetes.io/projected/c4637951-2a4e-4796-8b07-179acbdc0cfb-kube-api-access-gmdcr\") pod \"maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm\" (UID: \"c4637951-2a4e-4796-8b07-179acbdc0cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"
Nov 03 02:14:16.589988 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:16.589941    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4637951-2a4e-4796-8b07-179acbdc0cfb-maas-api-tls\") pod \"maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm\" (UID: \"c4637951-2a4e-4796-8b07-179acbdc0cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"
Nov 03 02:14:16.590108 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:16.590092    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4637951-2a4e-4796-8b07-179acbdc0cfb-openshift-service-ca\") pod \"maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm\" (UID: \"c4637951-2a4e-4796-8b07-179acbdc0cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"
Nov 03 02:14:16.691714 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:16.691660    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmdcr\" (UniqueName: \"kubernetes.io/projected/c4637951-2a4e-4796-8b07-179acbdc0cfb-kube-api-access-gmdcr\") pod \"maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm\" (UID: \"c4637951-2a4e-4796-8b07-179acbdc0cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"
Nov 03 02:14:16.691930 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:16.691784    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4637951-2a4e-4796-8b07-179acbdc0cfb-maas-api-tls\") pod \"maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm\" (UID: \"c4637951-2a4e-4796-8b07-179acbdc0cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"
Nov 03 02:14:16.691930 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:16.691892    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4637951-2a4e-4796-8b07-179acbdc0cfb-openshift-service-ca\") pod \"maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm\" (UID: \"c4637951-2a4e-4796-8b07-179acbdc0cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"
Nov 03 02:14:16.692055 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:14:16.691928    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-serving-cert: secret "maas-api-e2e-api-a-a772c3-serving-cert" not found
Nov 03 02:14:16.692055 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:16.691941    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4637951-2a4e-4796-8b07-179acbdc0cfb-metrics-certs\") pod \"maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm\" (UID: \"c4637951-2a4e-4796-8b07-179acbdc0cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"
Nov 03 02:14:16.692055 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:14:16.692006    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4637951-2a4e-4796-8b07-179acbdc0cfb-maas-api-tls podName:c4637951-2a4e-4796-8b07-179acbdc0cfb nodeName:}" failed. No retries permitted until 2026-11-03 02:14:17.191986321 +0000 UTC m=+1384.206227681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c4637951-2a4e-4796-8b07-179acbdc0cfb-maas-api-tls") pod "maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm" (UID: "c4637951-2a4e-4796-8b07-179acbdc0cfb") : secret "maas-api-e2e-api-a-a772c3-serving-cert" not found
Nov 03 02:14:16.692576 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:16.692553    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4637951-2a4e-4796-8b07-179acbdc0cfb-openshift-service-ca\") pod \"maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm\" (UID: \"c4637951-2a4e-4796-8b07-179acbdc0cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"
Nov 03 02:14:16.694421 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:16.694401    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4637951-2a4e-4796-8b07-179acbdc0cfb-metrics-certs\") pod \"maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm\" (UID: \"c4637951-2a4e-4796-8b07-179acbdc0cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"
Nov 03 02:14:16.746007 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:16.745930    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmdcr\" (UniqueName: \"kubernetes.io/projected/c4637951-2a4e-4796-8b07-179acbdc0cfb-kube-api-access-gmdcr\") pod \"maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm\" (UID: \"c4637951-2a4e-4796-8b07-179acbdc0cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"
Nov 03 02:14:17.200341 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:17.200294    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4637951-2a4e-4796-8b07-179acbdc0cfb-maas-api-tls\") pod \"maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm\" (UID: \"c4637951-2a4e-4796-8b07-179acbdc0cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"
Nov 03 02:14:17.203001 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:17.202971    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4637951-2a4e-4796-8b07-179acbdc0cfb-maas-api-tls\") pod \"maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm\" (UID: \"c4637951-2a4e-4796-8b07-179acbdc0cfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"
Nov 03 02:14:17.381138 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:17.381100    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"
Nov 03 02:14:17.757610 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:17.757575    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"]
Nov 03 02:14:17.762954 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:14:17.762926    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4637951_2a4e_4796_8b07_179acbdc0cfb.slice/crio-7c04174f37f9d795f2f95033312e3596dd571e50da36f7f785bb0e854b680b21 WatchSource:0}: Error finding container 7c04174f37f9d795f2f95033312e3596dd571e50da36f7f785bb0e854b680b21: Status 404 returned error can't find the container with id 7c04174f37f9d795f2f95033312e3596dd571e50da36f7f785bb0e854b680b21
Nov 03 02:14:18.696137 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:18.696107    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm" event={"ID":"c4637951-2a4e-4796-8b07-179acbdc0cfb","Type":"ContainerStarted","Data":"7c04174f37f9d795f2f95033312e3596dd571e50da36f7f785bb0e854b680b21"}
Nov 03 02:14:19.705097 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:19.705064    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm" event={"ID":"c4637951-2a4e-4796-8b07-179acbdc0cfb","Type":"ContainerStarted","Data":"fc5c2d6048b147da22eb51f174a5486c3d5cc374b3f25b75c423bf1044088e30"}
Nov 03 02:14:19.705563 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:19.705197    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"
Nov 03 02:14:19.812471 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:19.812404    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm" podStartSLOduration=2.94010362 podStartE2EDuration="3.812387495s" totalImagesPullingTime="872.283875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:14:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:14:19.811851967 +0000 UTC m=+1386.826093334" watchObservedRunningTime="2026-11-03 02:14:19.812387495 +0000 UTC m=+1386.826628864"
Nov 03 02:14:25.716845 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:25.716763    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"
Nov 03 02:14:30.733865 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:14:30.733811    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:14:35.963725 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:35.963673    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"]
Nov 03 02:14:35.968338 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:35.968313    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.017043 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.017005    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a0e72e-openshift-default-dockercfg-cskrh\""
Nov 03 02:14:36.069477 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.069430    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"]
Nov 03 02:14:36.069926 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.069562    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.069926 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.069641    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da7afda4-1bc9-418d-8a57-197aae00cb9e-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.069926 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.069872    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.070126 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.069960    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.070126 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.069986    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.070126 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.070023    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.070126 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.070050    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kbw8\" (UniqueName: \"kubernetes.io/projected/da7afda4-1bc9-418d-8a57-197aae00cb9e-kube-api-access-7kbw8\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.070126 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.070099    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.070368 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.070141    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.170944 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.170895    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.171148 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.171001    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.171148 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.171020    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.171148 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.171052    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.171148 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.171073    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kbw8\" (UniqueName: \"kubernetes.io/projected/da7afda4-1bc9-418d-8a57-197aae00cb9e-kube-api-access-7kbw8\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.171148 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.171124    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.171418 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.171394    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.171501 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.171481    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.171569 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.171542    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.171624 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.171591    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.171703 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.171670    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.171824 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.171801    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da7afda4-1bc9-418d-8a57-197aae00cb9e-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.171887 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.171867    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.172431 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.172408    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da7afda4-1bc9-418d-8a57-197aae00cb9e-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.173989 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.173963    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.174148 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.174072    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.224222 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.224132    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.224390 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.224266    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kbw8\" (UniqueName: \"kubernetes.io/projected/da7afda4-1bc9-418d-8a57-197aae00cb9e-kube-api-access-7kbw8\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.282629 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.282587    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:36.720382 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.720343    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"]
Nov 03 02:14:36.727546 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:14:36.727511    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7afda4_1bc9_418d_8a57_197aae00cb9e.slice/crio-f55529e0dc119e1404617f39dd3055b8d3506fc238993df1187f83cdf7508bec WatchSource:0}: Error finding container f55529e0dc119e1404617f39dd3055b8d3506fc238993df1187f83cdf7508bec: Status 404 returned error can't find the container with id f55529e0dc119e1404617f39dd3055b8d3506fc238993df1187f83cdf7508bec
Nov 03 02:14:36.730876 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.730832    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:14:36.730979 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.730936    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:14:36.731020 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.730982    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:14:36.840873 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.840839    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg" event={"ID":"da7afda4-1bc9-418d-8a57-197aae00cb9e","Type":"ContainerStarted","Data":"00899eb9b54d882e75fe598b93275b93413c34c56a8fec318ef63ef80eb0d523"}
Nov 03 02:14:36.841043 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.840899    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg" event={"ID":"da7afda4-1bc9-418d-8a57-197aae00cb9e","Type":"ContainerStarted","Data":"f55529e0dc119e1404617f39dd3055b8d3506fc238993df1187f83cdf7508bec"}
Nov 03 02:14:36.951012 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:36.950953    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg" podStartSLOduration=1.950936614 podStartE2EDuration="1.950936614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:14:36.950234387 +0000 UTC m=+1403.964475754" watchObservedRunningTime="2026-11-03 02:14:36.950936614 +0000 UTC m=+1403.965177959"
Nov 03 02:14:37.283598 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:37.283557    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:37.285153 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:37.285117    5005 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 02:14:37.285303 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:37.285178    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg" podUID="da7afda4-1bc9-418d-8a57-197aae00cb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 02:14:38.283459 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:38.283419    5005 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 02:14:38.283648 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:38.283485    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg" podUID="da7afda4-1bc9-418d-8a57-197aae00cb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 02:14:39.283084 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:39.283048    5005 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 02:14:39.283258 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:39.283103    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg" podUID="da7afda4-1bc9-418d-8a57-197aae00cb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 02:14:40.283912 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:40.283868    5005 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 02:14:40.284316 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:40.283933    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg" podUID="da7afda4-1bc9-418d-8a57-197aae00cb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 02:14:41.283153 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:41.283113    5005 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 02:14:41.283342 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:41.283170    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg" podUID="da7afda4-1bc9-418d-8a57-197aae00cb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 02:14:42.284055 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:42.284018    5005 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 02:14:42.284487 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:42.284076    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg" podUID="da7afda4-1bc9-418d-8a57-197aae00cb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 02:14:43.283614 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:43.283573    5005 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 02:14:43.283822 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:43.283645    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg" podUID="da7afda4-1bc9-418d-8a57-197aae00cb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 02:14:44.287988 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:44.287955    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:44.288500 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:44.288479    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:14:44.289129 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:14:44.289107    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:15:00.716133 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:15:00.716087    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:15:10.055726 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:10.055664    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"]
Nov 03 02:15:10.060427 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:10.060405    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"
Nov 03 02:15:10.109130 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:10.109103    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c617de-serving-cert\""
Nov 03 02:15:10.160134 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:10.160105    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"]
Nov 03 02:15:10.190307 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:10.190265    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b02bc9ee-d26b-48b2-b2ce-06784e778bce-metrics-certs\") pod \"maas-api-e2e-api-b-c617de-76d6557dbf-8hls7\" (UID: \"b02bc9ee-d26b-48b2-b2ce-06784e778bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"
Nov 03 02:15:10.193245 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:10.192608    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shv2g\" (UniqueName: \"kubernetes.io/projected/b02bc9ee-d26b-48b2-b2ce-06784e778bce-kube-api-access-shv2g\") pod \"maas-api-e2e-api-b-c617de-76d6557dbf-8hls7\" (UID: \"b02bc9ee-d26b-48b2-b2ce-06784e778bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"
Nov 03 02:15:10.193797 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:10.193765    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02bc9ee-d26b-48b2-b2ce-06784e778bce-openshift-service-ca\") pod \"maas-api-e2e-api-b-c617de-76d6557dbf-8hls7\" (UID: \"b02bc9ee-d26b-48b2-b2ce-06784e778bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"
Nov 03 02:15:10.194156 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:10.194131    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02bc9ee-d26b-48b2-b2ce-06784e778bce-maas-api-tls\") pod \"maas-api-e2e-api-b-c617de-76d6557dbf-8hls7\" (UID: \"b02bc9ee-d26b-48b2-b2ce-06784e778bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"
Nov 03 02:15:10.295746 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:10.295676    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02bc9ee-d26b-48b2-b2ce-06784e778bce-maas-api-tls\") pod \"maas-api-e2e-api-b-c617de-76d6557dbf-8hls7\" (UID: \"b02bc9ee-d26b-48b2-b2ce-06784e778bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"
Nov 03 02:15:10.296768 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:15:10.296066    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-serving-cert: secret "maas-api-e2e-api-b-c617de-serving-cert" not found
Nov 03 02:15:10.296995 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:15:10.296983    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b02bc9ee-d26b-48b2-b2ce-06784e778bce-maas-api-tls podName:b02bc9ee-d26b-48b2-b2ce-06784e778bce nodeName:}" failed. No retries permitted until 2026-11-03 02:15:10.796957434 +0000 UTC m=+1437.811198792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b02bc9ee-d26b-48b2-b2ce-06784e778bce-maas-api-tls") pod "maas-api-e2e-api-b-c617de-76d6557dbf-8hls7" (UID: "b02bc9ee-d26b-48b2-b2ce-06784e778bce") : secret "maas-api-e2e-api-b-c617de-serving-cert" not found
Nov 03 02:15:10.297418 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:10.297399    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b02bc9ee-d26b-48b2-b2ce-06784e778bce-metrics-certs\") pod \"maas-api-e2e-api-b-c617de-76d6557dbf-8hls7\" (UID: \"b02bc9ee-d26b-48b2-b2ce-06784e778bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"
Nov 03 02:15:10.298121 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:10.298100    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shv2g\" (UniqueName: \"kubernetes.io/projected/b02bc9ee-d26b-48b2-b2ce-06784e778bce-kube-api-access-shv2g\") pod \"maas-api-e2e-api-b-c617de-76d6557dbf-8hls7\" (UID: \"b02bc9ee-d26b-48b2-b2ce-06784e778bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"
Nov 03 02:15:10.299139 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:10.299109    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02bc9ee-d26b-48b2-b2ce-06784e778bce-openshift-service-ca\") pod \"maas-api-e2e-api-b-c617de-76d6557dbf-8hls7\" (UID: \"b02bc9ee-d26b-48b2-b2ce-06784e778bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"
Nov 03 02:15:10.299980 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:10.299954    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02bc9ee-d26b-48b2-b2ce-06784e778bce-openshift-service-ca\") pod \"maas-api-e2e-api-b-c617de-76d6557dbf-8hls7\" (UID: \"b02bc9ee-d26b-48b2-b2ce-06784e778bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"
Nov 03 02:15:10.302013 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:10.301990    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b02bc9ee-d26b-48b2-b2ce-06784e778bce-metrics-certs\") pod \"maas-api-e2e-api-b-c617de-76d6557dbf-8hls7\" (UID: \"b02bc9ee-d26b-48b2-b2ce-06784e778bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"
Nov 03 02:15:10.355015 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:10.354942    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shv2g\" (UniqueName: \"kubernetes.io/projected/b02bc9ee-d26b-48b2-b2ce-06784e778bce-kube-api-access-shv2g\") pod \"maas-api-e2e-api-b-c617de-76d6557dbf-8hls7\" (UID: \"b02bc9ee-d26b-48b2-b2ce-06784e778bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"
Nov 03 02:15:10.808596 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:10.808557    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02bc9ee-d26b-48b2-b2ce-06784e778bce-maas-api-tls\") pod \"maas-api-e2e-api-b-c617de-76d6557dbf-8hls7\" (UID: \"b02bc9ee-d26b-48b2-b2ce-06784e778bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"
Nov 03 02:15:10.811059 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:10.811032    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02bc9ee-d26b-48b2-b2ce-06784e778bce-maas-api-tls\") pod \"maas-api-e2e-api-b-c617de-76d6557dbf-8hls7\" (UID: \"b02bc9ee-d26b-48b2-b2ce-06784e778bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"
Nov 03 02:15:10.975449 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:10.975411    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"
Nov 03 02:15:11.413262 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:11.413229    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"]
Nov 03 02:15:11.419018 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:15:11.418989    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02bc9ee_d26b_48b2_b2ce_06784e778bce.slice/crio-372d6bab490cc25131ff20baed61ad3ee98e6d6ecedef2a600439e5dd2b1ccd1 WatchSource:0}: Error finding container 372d6bab490cc25131ff20baed61ad3ee98e6d6ecedef2a600439e5dd2b1ccd1: Status 404 returned error can't find the container with id 372d6bab490cc25131ff20baed61ad3ee98e6d6ecedef2a600439e5dd2b1ccd1
Nov 03 02:15:12.142014 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:12.141976    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7" event={"ID":"b02bc9ee-d26b-48b2-b2ce-06784e778bce","Type":"ContainerStarted","Data":"372d6bab490cc25131ff20baed61ad3ee98e6d6ecedef2a600439e5dd2b1ccd1"}
Nov 03 02:15:13.154295 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:13.153218    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7" event={"ID":"b02bc9ee-d26b-48b2-b2ce-06784e778bce","Type":"ContainerStarted","Data":"fb89e970b0de336b2b57d20e9b0053cf123e3f1406057ec3f0b13ebc9477a007"}
Nov 03 02:15:13.154295 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:13.153842    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"
Nov 03 02:15:13.264901 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:13.264841    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7" podStartSLOduration=2.3929403430000002 podStartE2EDuration="3.264826319s" totalImagesPullingTime="871.885976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:13.261838601 +0000 UTC m=+1440.276079967" watchObservedRunningTime="2026-11-03 02:15:13.264826319 +0000 UTC m=+1440.279067685"
Nov 03 02:15:19.168555 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:19.168513    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"
Nov 03 02:15:30.717620 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:15:30.717517    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:15:36.415116 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.415079    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"]
Nov 03 02:15:36.419716 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.419682    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.557397 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.557359    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.557588 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.557417    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.557588 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.557524    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rblt\" (UniqueName: \"kubernetes.io/projected/8e19ed57-84d2-499a-8a62-3b5beac110a2-kube-api-access-7rblt\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.557588 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.557564    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.557588 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.557583    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.557873 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.557828    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.557873 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.557853    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e19ed57-84d2-499a-8a62-3b5beac110a2-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.557990 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.557872    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.557990 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.557906    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.615476 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.615442    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"]
Nov 03 02:15:36.659214 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.659169    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.659420 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.659278    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rblt\" (UniqueName: \"kubernetes.io/projected/8e19ed57-84d2-499a-8a62-3b5beac110a2-kube-api-access-7rblt\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.659420 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.659335    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.659420 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.659363    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.659608 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.659556    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.659608 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.659585    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e19ed57-84d2-499a-8a62-3b5beac110a2-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.659742 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.659614    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.659742 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.659643    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-workload-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.659742 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.659649    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.659896 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.659786    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.659991 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.659970    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-data\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.660381 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.660359    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-workload-certs\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.660987 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.660960    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e19ed57-84d2-499a-8a62-3b5beac110a2-istiod-ca-cert\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.661095 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.660533    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-credential-socket\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.663450 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.663423    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-envoy\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.663675 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.663655    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-podinfo\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.719854 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.719738    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-token\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.721635 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.721593    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rblt\" (UniqueName: \"kubernetes.io/projected/8e19ed57-84d2-499a-8a62-3b5beac110a2-kube-api-access-7rblt\") pod \"e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") " pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:36.776444 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:36.776394    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:37.245285 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:37.245231    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"]
Nov 03 02:15:37.348236 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:37.348181    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:15:37.348390 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:37.348310    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:15:37.348390 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:37.348373    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:15:37.371504 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:37.371461    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc" event={"ID":"8e19ed57-84d2-499a-8a62-3b5beac110a2","Type":"ContainerStarted","Data":"77ae09c334b847c8edb47a582034529333a79fde8e1b0496a96f44b20ddcd5dc"}
Nov 03 02:15:38.381792 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:38.381740    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc" event={"ID":"8e19ed57-84d2-499a-8a62-3b5beac110a2","Type":"ContainerStarted","Data":"826c755b33d9929f5ae76165cee379d8245ab5b20a824900a11c118fa56a698e"}
Nov 03 02:15:38.495937 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:38.495862    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc" podStartSLOduration=2.495840836 podStartE2EDuration="2.495840836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:38.491543031 +0000 UTC m=+1465.505784400" watchObservedRunningTime="2026-11-03 02:15:38.495840836 +0000 UTC m=+1465.510082218"
Nov 03 02:15:38.778058 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:38.778015    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:38.779730 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:38.779690    5005 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 02:15:38.779877 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:38.779759    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc" podUID="8e19ed57-84d2-499a-8a62-3b5beac110a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 02:15:39.778198 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:39.778160    5005 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 02:15:39.778632 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:39.778227    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc" podUID="8e19ed57-84d2-499a-8a62-3b5beac110a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 02:15:40.779251 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:40.779208    5005 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 02:15:40.779845 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:40.779274    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc" podUID="8e19ed57-84d2-499a-8a62-3b5beac110a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 02:15:41.779096 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:41.779051    5005 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 02:15:41.779302 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:41.779123    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc" podUID="8e19ed57-84d2-499a-8a62-3b5beac110a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 02:15:42.778952 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:42.778891    5005 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 02:15:42.779220 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:42.778983    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc" podUID="8e19ed57-84d2-499a-8a62-3b5beac110a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 02:15:43.778996 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:43.778953    5005 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 03 02:15:43.779535 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:43.779013    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc" podUID="8e19ed57-84d2-499a-8a62-3b5beac110a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 03 02:15:44.782975 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:44.782930    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd"]
Nov 03 02:15:44.783661 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:44.783640    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:44.783939 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:44.783918    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:44.784754 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:44.784722    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:15:44.888407 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:44.888365    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst"]
Nov 03 02:15:49.783723 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:49.783659    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd" podUID="4a3cc135-6421-48ed-83ab-3c9c8944d191" containerName="payload-processing" containerID="cri-o://648e571f10a3caf36440ca8badc163b8c7ff05fd2c448882db73979f10e090b9" gracePeriod=25
Nov 03 02:15:49.889826 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:49.889761    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst" podUID="e520cfcb-166e-469a-ba57-739da7f713da" containerName="payload-pre-processing" containerID="cri-o://4d1fca288a73ed220f67ed6a78b67920934f421cee33badb58cfd8ec9be3c40b" gracePeriod=25
Nov 03 02:15:51.327237 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:51.327195    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd" podUID="4a3cc135-6421-48ed-83ab-3c9c8944d191" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:15:51.327646 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:15:51.327324    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst" podUID="e520cfcb-166e-469a-ba57-739da7f713da" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:16:00.714984 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:16:00.714894    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:16:01.327969 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:01.327930    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd" podUID="4a3cc135-6421-48ed-83ab-3c9c8944d191" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:16:01.328195 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:01.327930    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst" podUID="e520cfcb-166e-469a-ba57-739da7f713da" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:16:03.666360 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:03.666326    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"]
Nov 03 02:16:03.671330 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:03.671302    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"
Nov 03 02:16:03.717890 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:03.717853    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8fac4d-a231-421f-993e-981c21ca4e81-openshift-service-ca\") pod \"maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w\" (UID: \"1f8fac4d-a231-421f-993e-981c21ca4e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"
Nov 03 02:16:03.717890 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:03.717893    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh7fr\" (UniqueName: \"kubernetes.io/projected/1f8fac4d-a231-421f-993e-981c21ca4e81-kube-api-access-rh7fr\") pod \"maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w\" (UID: \"1f8fac4d-a231-421f-993e-981c21ca4e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"
Nov 03 02:16:03.718106 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:03.718084    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f8fac4d-a231-421f-993e-981c21ca4e81-metrics-certs\") pod \"maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w\" (UID: \"1f8fac4d-a231-421f-993e-981c21ca4e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"
Nov 03 02:16:03.718146 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:03.718137    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f8fac4d-a231-421f-993e-981c21ca4e81-maas-api-tls\") pod \"maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w\" (UID: \"1f8fac4d-a231-421f-993e-981c21ca4e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"
Nov 03 02:16:03.724639 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:03.724608    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"]
Nov 03 02:16:03.766040 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:03.766005    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-aa82d2-serving-cert\""
Nov 03 02:16:03.819419 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:03.819382    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8fac4d-a231-421f-993e-981c21ca4e81-openshift-service-ca\") pod \"maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w\" (UID: \"1f8fac4d-a231-421f-993e-981c21ca4e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"
Nov 03 02:16:03.819601 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:03.819433    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh7fr\" (UniqueName: \"kubernetes.io/projected/1f8fac4d-a231-421f-993e-981c21ca4e81-kube-api-access-rh7fr\") pod \"maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w\" (UID: \"1f8fac4d-a231-421f-993e-981c21ca4e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"
Nov 03 02:16:03.819601 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:03.819564    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f8fac4d-a231-421f-993e-981c21ca4e81-metrics-certs\") pod \"maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w\" (UID: \"1f8fac4d-a231-421f-993e-981c21ca4e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"
Nov 03 02:16:03.819782 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:03.819629    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f8fac4d-a231-421f-993e-981c21ca4e81-maas-api-tls\") pod \"maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w\" (UID: \"1f8fac4d-a231-421f-993e-981c21ca4e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"
Nov 03 02:16:03.819901 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:16:03.819879    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-serving-cert: secret "maas-api-e2e-api-b-aa82d2-serving-cert" not found
Nov 03 02:16:03.819994 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:16:03.819982    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f8fac4d-a231-421f-993e-981c21ca4e81-maas-api-tls podName:1f8fac4d-a231-421f-993e-981c21ca4e81 nodeName:}" failed. No retries permitted until 2026-11-03 02:16:04.319958617 +0000 UTC m=+1491.334199965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1f8fac4d-a231-421f-993e-981c21ca4e81-maas-api-tls") pod "maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w" (UID: "1f8fac4d-a231-421f-993e-981c21ca4e81") : secret "maas-api-e2e-api-b-aa82d2-serving-cert" not found
Nov 03 02:16:03.820312 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:03.820281    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8fac4d-a231-421f-993e-981c21ca4e81-openshift-service-ca\") pod \"maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w\" (UID: \"1f8fac4d-a231-421f-993e-981c21ca4e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"
Nov 03 02:16:03.830619 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:03.830585    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f8fac4d-a231-421f-993e-981c21ca4e81-metrics-certs\") pod \"maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w\" (UID: \"1f8fac4d-a231-421f-993e-981c21ca4e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"
Nov 03 02:16:03.874831 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:03.874797    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh7fr\" (UniqueName: \"kubernetes.io/projected/1f8fac4d-a231-421f-993e-981c21ca4e81-kube-api-access-rh7fr\") pod \"maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w\" (UID: \"1f8fac4d-a231-421f-993e-981c21ca4e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"
Nov 03 02:16:04.330198 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:04.330160    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f8fac4d-a231-421f-993e-981c21ca4e81-maas-api-tls\") pod \"maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w\" (UID: \"1f8fac4d-a231-421f-993e-981c21ca4e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"
Nov 03 02:16:04.332683 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:04.332657    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f8fac4d-a231-421f-993e-981c21ca4e81-maas-api-tls\") pod \"maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w\" (UID: \"1f8fac4d-a231-421f-993e-981c21ca4e81\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"
Nov 03 02:16:04.587088 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:04.586999    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"
Nov 03 02:16:04.964000 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:04.963921    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"]
Nov 03 02:16:04.969028 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:16:04.968997    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8fac4d_a231_421f_993e_981c21ca4e81.slice/crio-73bbacc78d3efe1ddeff408a722e4d7daabe1e5f0057d2a2f2551af2c917280a WatchSource:0}: Error finding container 73bbacc78d3efe1ddeff408a722e4d7daabe1e5f0057d2a2f2551af2c917280a: Status 404 returned error can't find the container with id 73bbacc78d3efe1ddeff408a722e4d7daabe1e5f0057d2a2f2551af2c917280a
Nov 03 02:16:05.612019 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:05.611977    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w" event={"ID":"1f8fac4d-a231-421f-993e-981c21ca4e81","Type":"ContainerStarted","Data":"73bbacc78d3efe1ddeff408a722e4d7daabe1e5f0057d2a2f2551af2c917280a"}
Nov 03 02:16:06.620228 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:06.620191    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w" event={"ID":"1f8fac4d-a231-421f-993e-981c21ca4e81","Type":"ContainerStarted","Data":"9cf45661c6134ef66262196379a2bf457d0112205b3121d2dacf7bc1f83f28fe"}
Nov 03 02:16:06.620620 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:06.620293    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"
Nov 03 02:16:06.727743 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:06.727652    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w" podStartSLOduration=2.886832731 podStartE2EDuration="3.727636004s" totalImagesPullingTime="840.803273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:16:06.726454239 +0000 UTC m=+1493.740695605" watchObservedRunningTime="2026-11-03 02:16:06.727636004 +0000 UTC m=+1493.741877372"
Nov 03 02:16:10.092126 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.091750    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd"
Nov 03 02:16:10.176759 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.176732    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst"
Nov 03 02:16:10.224950 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.224919    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e520cfcb-166e-469a-ba57-739da7f713da-kube-api-access-47xb2\" (UniqueName: \"kubernetes.io/projected/e520cfcb-166e-469a-ba57-739da7f713da-kube-api-access-47xb2\") pod \"e520cfcb-166e-469a-ba57-739da7f713da\" (UID: \"e520cfcb-166e-469a-ba57-739da7f713da\") "
Nov 03 02:16:10.225170 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.224968    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a3cc135-6421-48ed-83ab-3c9c8944d191-kube-api-access-ncmt9\" (UniqueName: \"kubernetes.io/projected/4a3cc135-6421-48ed-83ab-3c9c8944d191-kube-api-access-ncmt9\") pod \"4a3cc135-6421-48ed-83ab-3c9c8944d191\" (UID: \"4a3cc135-6421-48ed-83ab-3c9c8944d191\") "
Nov 03 02:16:10.225170 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.225081    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e520cfcb-166e-469a-ba57-739da7f713da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e520cfcb-166e-469a-ba57-739da7f713da-plugins-config-volume\") pod \"e520cfcb-166e-469a-ba57-739da7f713da\" (UID: \"e520cfcb-166e-469a-ba57-739da7f713da\") "
Nov 03 02:16:10.225306 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.225176    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a3cc135-6421-48ed-83ab-3c9c8944d191-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a3cc135-6421-48ed-83ab-3c9c8944d191-plugins-config-volume\") pod \"4a3cc135-6421-48ed-83ab-3c9c8944d191\" (UID: \"4a3cc135-6421-48ed-83ab-3c9c8944d191\") "
Nov 03 02:16:10.225561 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.225529    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e520cfcb-166e-469a-ba57-739da7f713da-plugins-config-volume" pod "e520cfcb-166e-469a-ba57-739da7f713da" (UID: "e520cfcb-166e-469a-ba57-739da7f713da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:16:10.225807 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.225788    5005 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e520cfcb-166e-469a-ba57-739da7f713da-plugins-config-volume\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:10.226043 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.226001    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3cc135-6421-48ed-83ab-3c9c8944d191-plugins-config-volume" pod "4a3cc135-6421-48ed-83ab-3c9c8944d191" (UID: "4a3cc135-6421-48ed-83ab-3c9c8944d191"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:16:10.228155 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.228131    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3cc135-6421-48ed-83ab-3c9c8944d191-kube-api-access-ncmt9" pod "4a3cc135-6421-48ed-83ab-3c9c8944d191" (UID: "4a3cc135-6421-48ed-83ab-3c9c8944d191"). InnerVolumeSpecName "kube-api-access-ncmt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:16:10.228437 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.228409    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e520cfcb-166e-469a-ba57-739da7f713da-kube-api-access-47xb2" pod "e520cfcb-166e-469a-ba57-739da7f713da" (UID: "e520cfcb-166e-469a-ba57-739da7f713da"). InnerVolumeSpecName "kube-api-access-47xb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:16:10.327243 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.327144    5005 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a3cc135-6421-48ed-83ab-3c9c8944d191-plugins-config-volume\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:10.327243 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.327185    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47xb2\" (UniqueName: \"kubernetes.io/projected/e520cfcb-166e-469a-ba57-739da7f713da-kube-api-access-47xb2\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:10.327243 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.327201    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncmt9\" (UniqueName: \"kubernetes.io/projected/4a3cc135-6421-48ed-83ab-3c9c8944d191-kube-api-access-ncmt9\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:10.654224 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.654135    5005 generic.go:356] "Generic (PLEG): container finished" podID="4a3cc135-6421-48ed-83ab-3c9c8944d191" containerID="648e571f10a3caf36440ca8badc163b8c7ff05fd2c448882db73979f10e090b9" exitCode=0
Nov 03 02:16:10.654224 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.654180    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd"
Nov 03 02:16:10.654224 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.654212    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd" event={"ID":"4a3cc135-6421-48ed-83ab-3c9c8944d191","Type":"ContainerDied","Data":"648e571f10a3caf36440ca8badc163b8c7ff05fd2c448882db73979f10e090b9"}
Nov 03 02:16:10.654480 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.654271    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd" event={"ID":"4a3cc135-6421-48ed-83ab-3c9c8944d191","Type":"ContainerDied","Data":"96b9d2bb341fd312d7e85b8c6315845067ddb2ff022bdb79b608b94c88db8fc8"}
Nov 03 02:16:10.654480 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.654293    5005 scope.go:122] "RemoveContainer" containerID="648e571f10a3caf36440ca8badc163b8c7ff05fd2c448882db73979f10e090b9"
Nov 03 02:16:10.655934 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.655906    5005 generic.go:356] "Generic (PLEG): container finished" podID="e520cfcb-166e-469a-ba57-739da7f713da" containerID="4d1fca288a73ed220f67ed6a78b67920934f421cee33badb58cfd8ec9be3c40b" exitCode=0
Nov 03 02:16:10.656083 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.655954    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst"
Nov 03 02:16:10.656083 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.655957    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst" event={"ID":"e520cfcb-166e-469a-ba57-739da7f713da","Type":"ContainerDied","Data":"4d1fca288a73ed220f67ed6a78b67920934f421cee33badb58cfd8ec9be3c40b"}
Nov 03 02:16:10.656194 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.656091    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst" event={"ID":"e520cfcb-166e-469a-ba57-739da7f713da","Type":"ContainerDied","Data":"b9ad72cc0d83ed73e83f061e532ef868d6780623053bb96b7b26f56fd924c426"}
Nov 03 02:16:10.665285 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.665265    5005 scope.go:122] "RemoveContainer" containerID="648e571f10a3caf36440ca8badc163b8c7ff05fd2c448882db73979f10e090b9"
Nov 03 02:16:10.665553 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:16:10.665528    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"648e571f10a3caf36440ca8badc163b8c7ff05fd2c448882db73979f10e090b9\": container with ID starting with 648e571f10a3caf36440ca8badc163b8c7ff05fd2c448882db73979f10e090b9 not found: ID does not exist" containerID="648e571f10a3caf36440ca8badc163b8c7ff05fd2c448882db73979f10e090b9"
Nov 03 02:16:10.665605 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.665568    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648e571f10a3caf36440ca8badc163b8c7ff05fd2c448882db73979f10e090b9"} err="failed to get container status \"648e571f10a3caf36440ca8badc163b8c7ff05fd2c448882db73979f10e090b9\": rpc error: code = NotFound desc = could not find container \"648e571f10a3caf36440ca8badc163b8c7ff05fd2c448882db73979f10e090b9\": container with ID starting with 648e571f10a3caf36440ca8badc163b8c7ff05fd2c448882db73979f10e090b9 not found: ID does not exist"
Nov 03 02:16:10.665605 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.665593    5005 scope.go:122] "RemoveContainer" containerID="4d1fca288a73ed220f67ed6a78b67920934f421cee33badb58cfd8ec9be3c40b"
Nov 03 02:16:10.675960 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.675940    5005 scope.go:122] "RemoveContainer" containerID="4d1fca288a73ed220f67ed6a78b67920934f421cee33badb58cfd8ec9be3c40b"
Nov 03 02:16:10.676220 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:16:10.676201    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d1fca288a73ed220f67ed6a78b67920934f421cee33badb58cfd8ec9be3c40b\": container with ID starting with 4d1fca288a73ed220f67ed6a78b67920934f421cee33badb58cfd8ec9be3c40b not found: ID does not exist" containerID="4d1fca288a73ed220f67ed6a78b67920934f421cee33badb58cfd8ec9be3c40b"
Nov 03 02:16:10.676299 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.676228    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d1fca288a73ed220f67ed6a78b67920934f421cee33badb58cfd8ec9be3c40b"} err="failed to get container status \"4d1fca288a73ed220f67ed6a78b67920934f421cee33badb58cfd8ec9be3c40b\": rpc error: code = NotFound desc = could not find container \"4d1fca288a73ed220f67ed6a78b67920934f421cee33badb58cfd8ec9be3c40b\": container with ID starting with 4d1fca288a73ed220f67ed6a78b67920934f421cee33badb58cfd8ec9be3c40b not found: ID does not exist"
Nov 03 02:16:10.809013 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.808984    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd"]
Nov 03 02:16:10.812597 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.812575    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-701ba30f-54bcbfd7cf-ncgtd"]
Nov 03 02:16:10.914380 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.914291    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst"]
Nov 03 02:16:10.918000 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:10.917978    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-701ba30f-d59b89cf9-hlqst"]
Nov 03 02:16:11.110990 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:11.110957    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a3cc135-6421-48ed-83ab-3c9c8944d191" path="/var/lib/kubelet/pods/4a3cc135-6421-48ed-83ab-3c9c8944d191/volumes"
Nov 03 02:16:11.111468 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:11.111454    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e520cfcb-166e-469a-ba57-739da7f713da" path="/var/lib/kubelet/pods/e520cfcb-166e-469a-ba57-739da7f713da/volumes"
Nov 03 02:16:12.636842 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:12.636804    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"
Nov 03 02:16:13.308763 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:13.308733    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:16:13.315506 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:13.315471    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:16:23.758925 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:23.758890    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"]
Nov 03 02:16:23.759681 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:23.759662    5005 cpu_manager.go:414] "containerMap: removing container" podUID="4a3cc135-6421-48ed-83ab-3c9c8944d191" containerName="payload-processing"
Nov 03 02:16:23.759806 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:23.759684    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a3cc135-6421-48ed-83ab-3c9c8944d191" containerName="payload-processing"
Nov 03 02:16:23.759806 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:23.759728    5005 cpu_manager.go:414] "containerMap: removing container" podUID="e520cfcb-166e-469a-ba57-739da7f713da" containerName="payload-pre-processing"
Nov 03 02:16:23.759806 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:23.759734    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e520cfcb-166e-469a-ba57-739da7f713da" containerName="payload-pre-processing"
Nov 03 02:16:23.759972 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:23.759872    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a3cc135-6421-48ed-83ab-3c9c8944d191" containerName="payload-processing"
Nov 03 02:16:23.759972 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:23.759884    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="e520cfcb-166e-469a-ba57-739da7f713da" containerName="payload-pre-processing"
Nov 03 02:16:23.765391 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:23.765367    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"
Nov 03 02:16:23.829005 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:23.828967    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-6198c6-serving-cert\""
Nov 03 02:16:23.834992 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:23.834924    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0a3a84a-fd80-408e-854c-46f6f268c6fa-maas-api-tls\") pod \"maas-api-e2e-api-a-6198c6-75774b485c-swdtn\" (UID: \"c0a3a84a-fd80-408e-854c-46f6f268c6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"
Nov 03 02:16:23.839982 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:23.839939    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a3a84a-fd80-408e-854c-46f6f268c6fa-metrics-certs\") pod \"maas-api-e2e-api-a-6198c6-75774b485c-swdtn\" (UID: \"c0a3a84a-fd80-408e-854c-46f6f268c6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"
Nov 03 02:16:23.840266 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:23.840233    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a3a84a-fd80-408e-854c-46f6f268c6fa-openshift-service-ca\") pod \"maas-api-e2e-api-a-6198c6-75774b485c-swdtn\" (UID: \"c0a3a84a-fd80-408e-854c-46f6f268c6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"
Nov 03 02:16:23.840390 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:23.840297    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch6tl\" (UniqueName: \"kubernetes.io/projected/c0a3a84a-fd80-408e-854c-46f6f268c6fa-kube-api-access-ch6tl\") pod \"maas-api-e2e-api-a-6198c6-75774b485c-swdtn\" (UID: \"c0a3a84a-fd80-408e-854c-46f6f268c6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"
Nov 03 02:16:23.897183 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:23.897153    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"]
Nov 03 02:16:23.942753 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:23.942306    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0a3a84a-fd80-408e-854c-46f6f268c6fa-maas-api-tls\") pod \"maas-api-e2e-api-a-6198c6-75774b485c-swdtn\" (UID: \"c0a3a84a-fd80-408e-854c-46f6f268c6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"
Nov 03 02:16:23.942753 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:23.942428    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a3a84a-fd80-408e-854c-46f6f268c6fa-metrics-certs\") pod \"maas-api-e2e-api-a-6198c6-75774b485c-swdtn\" (UID: \"c0a3a84a-fd80-408e-854c-46f6f268c6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"
Nov 03 02:16:23.942753 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:23.942484    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a3a84a-fd80-408e-854c-46f6f268c6fa-openshift-service-ca\") pod \"maas-api-e2e-api-a-6198c6-75774b485c-swdtn\" (UID: \"c0a3a84a-fd80-408e-854c-46f6f268c6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"
Nov 03 02:16:23.942753 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:23.942530    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch6tl\" (UniqueName: \"kubernetes.io/projected/c0a3a84a-fd80-408e-854c-46f6f268c6fa-kube-api-access-ch6tl\") pod \"maas-api-e2e-api-a-6198c6-75774b485c-swdtn\" (UID: \"c0a3a84a-fd80-408e-854c-46f6f268c6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"
Nov 03 02:16:23.943207 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:16:23.943183    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-serving-cert: secret "maas-api-e2e-api-a-6198c6-serving-cert" not found
Nov 03 02:16:23.943281 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:16:23.943267    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a3a84a-fd80-408e-854c-46f6f268c6fa-maas-api-tls podName:c0a3a84a-fd80-408e-854c-46f6f268c6fa nodeName:}" failed. No retries permitted until 2026-11-03 02:16:24.443247188 +0000 UTC m=+1511.457488533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c0a3a84a-fd80-408e-854c-46f6f268c6fa-maas-api-tls") pod "maas-api-e2e-api-a-6198c6-75774b485c-swdtn" (UID: "c0a3a84a-fd80-408e-854c-46f6f268c6fa") : secret "maas-api-e2e-api-a-6198c6-serving-cert" not found
Nov 03 02:16:23.955677 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:23.955598    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a3a84a-fd80-408e-854c-46f6f268c6fa-openshift-service-ca\") pod \"maas-api-e2e-api-a-6198c6-75774b485c-swdtn\" (UID: \"c0a3a84a-fd80-408e-854c-46f6f268c6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"
Nov 03 02:16:23.969882 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:23.969844    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a3a84a-fd80-408e-854c-46f6f268c6fa-metrics-certs\") pod \"maas-api-e2e-api-a-6198c6-75774b485c-swdtn\" (UID: \"c0a3a84a-fd80-408e-854c-46f6f268c6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"
Nov 03 02:16:23.998760 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:23.998726    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch6tl\" (UniqueName: \"kubernetes.io/projected/c0a3a84a-fd80-408e-854c-46f6f268c6fa-kube-api-access-ch6tl\") pod \"maas-api-e2e-api-a-6198c6-75774b485c-swdtn\" (UID: \"c0a3a84a-fd80-408e-854c-46f6f268c6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"
Nov 03 02:16:24.448447 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:24.448408    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0a3a84a-fd80-408e-854c-46f6f268c6fa-maas-api-tls\") pod \"maas-api-e2e-api-a-6198c6-75774b485c-swdtn\" (UID: \"c0a3a84a-fd80-408e-854c-46f6f268c6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"
Nov 03 02:16:24.451101 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:24.451070    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0a3a84a-fd80-408e-854c-46f6f268c6fa-maas-api-tls\") pod \"maas-api-e2e-api-a-6198c6-75774b485c-swdtn\" (UID: \"c0a3a84a-fd80-408e-854c-46f6f268c6fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"
Nov 03 02:16:24.679564 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:24.679534    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"
Nov 03 02:16:25.056646 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:25.056606    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"]
Nov 03 02:16:25.061456 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:16:25.061422    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a3a84a_fd80_408e_854c_46f6f268c6fa.slice/crio-7639536a0d668cafd1f3c3ec45bd68eb602412601b87d24e0c0d82ad8d8ebd4d WatchSource:0}: Error finding container 7639536a0d668cafd1f3c3ec45bd68eb602412601b87d24e0c0d82ad8d8ebd4d: Status 404 returned error can't find the container with id 7639536a0d668cafd1f3c3ec45bd68eb602412601b87d24e0c0d82ad8d8ebd4d
Nov 03 02:16:25.784629 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:25.784595    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn" event={"ID":"c0a3a84a-fd80-408e-854c-46f6f268c6fa","Type":"ContainerStarted","Data":"7639536a0d668cafd1f3c3ec45bd68eb602412601b87d24e0c0d82ad8d8ebd4d"}
Nov 03 02:16:26.801282 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:26.801243    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn" event={"ID":"c0a3a84a-fd80-408e-854c-46f6f268c6fa","Type":"ContainerStarted","Data":"88ea407d596d64c8f336e8ba3e244b49a52c2e065d91377d3f34b90b5c21304f"}
Nov 03 02:16:26.801840 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:26.801345    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"
Nov 03 02:16:26.919743 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:26.915666    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn" podStartSLOduration=2.882489931 podStartE2EDuration="3.91564592s" totalImagesPullingTime="1.033155989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:16:26.912183625 +0000 UTC m=+1513.926424992" watchObservedRunningTime="2026-11-03 02:16:26.91564592 +0000 UTC m=+1513.929887288"
Nov 03 02:16:30.715390 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:16:30.715338    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:16:32.813752 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:32.813721    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"
Nov 03 02:16:37.783227 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:37.783187    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"]
Nov 03 02:16:37.783666 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:37.783403    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt" podUID="50e0635a-54b1-497e-9e2c-66f6961eee54" containerName="maas-api" containerID="cri-o://3741efb97dad30399eb4d28229b7f2d022eb18c1c74bea83ffd216321577d308" gracePeriod=30
Nov 03 02:16:38.083211 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:38.083180    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"
Nov 03 02:16:38.257949 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:38.257908    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50e0635a-54b1-497e-9e2c-66f6961eee54-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e0635a-54b1-497e-9e2c-66f6961eee54-openshift-service-ca\") pod \"50e0635a-54b1-497e-9e2c-66f6961eee54\" (UID: \"50e0635a-54b1-497e-9e2c-66f6961eee54\") "
Nov 03 02:16:38.258157 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:38.258058    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50e0635a-54b1-497e-9e2c-66f6961eee54-kube-api-access-fgs6v\" (UniqueName: \"kubernetes.io/projected/50e0635a-54b1-497e-9e2c-66f6961eee54-kube-api-access-fgs6v\") pod \"50e0635a-54b1-497e-9e2c-66f6961eee54\" (UID: \"50e0635a-54b1-497e-9e2c-66f6961eee54\") "
Nov 03 02:16:38.258969 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:38.258534    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50e0635a-54b1-497e-9e2c-66f6961eee54-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e0635a-54b1-497e-9e2c-66f6961eee54-metrics-certs\") pod \"50e0635a-54b1-497e-9e2c-66f6961eee54\" (UID: \"50e0635a-54b1-497e-9e2c-66f6961eee54\") "
Nov 03 02:16:38.258969 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:38.258590    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50e0635a-54b1-497e-9e2c-66f6961eee54-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50e0635a-54b1-497e-9e2c-66f6961eee54-maas-api-tls\") pod \"50e0635a-54b1-497e-9e2c-66f6961eee54\" (UID: \"50e0635a-54b1-497e-9e2c-66f6961eee54\") "
Nov 03 02:16:38.258969 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:38.258410    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e0635a-54b1-497e-9e2c-66f6961eee54-openshift-service-ca" pod "50e0635a-54b1-497e-9e2c-66f6961eee54" (UID: "50e0635a-54b1-497e-9e2c-66f6961eee54"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:16:38.260299 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:38.260270    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50e0635a-54b1-497e-9e2c-66f6961eee54-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:38.263003 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:38.262957    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e0635a-54b1-497e-9e2c-66f6961eee54-kube-api-access-fgs6v" pod "50e0635a-54b1-497e-9e2c-66f6961eee54" (UID: "50e0635a-54b1-497e-9e2c-66f6961eee54"). InnerVolumeSpecName "kube-api-access-fgs6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:16:38.266134 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:38.266091    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50e0635a-54b1-497e-9e2c-66f6961eee54-metrics-certs" pod "50e0635a-54b1-497e-9e2c-66f6961eee54" (UID: "50e0635a-54b1-497e-9e2c-66f6961eee54"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:16:38.266226 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:38.266139    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50e0635a-54b1-497e-9e2c-66f6961eee54-maas-api-tls" pod "50e0635a-54b1-497e-9e2c-66f6961eee54" (UID: "50e0635a-54b1-497e-9e2c-66f6961eee54"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:16:38.371110 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:38.370968    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgs6v\" (UniqueName: \"kubernetes.io/projected/50e0635a-54b1-497e-9e2c-66f6961eee54-kube-api-access-fgs6v\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:38.371110 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:38.371014    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50e0635a-54b1-497e-9e2c-66f6961eee54-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:38.371110 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:38.371030    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50e0635a-54b1-497e-9e2c-66f6961eee54-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:38.897433 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:38.897395    5005 generic.go:356] "Generic (PLEG): container finished" podID="50e0635a-54b1-497e-9e2c-66f6961eee54" containerID="3741efb97dad30399eb4d28229b7f2d022eb18c1c74bea83ffd216321577d308" exitCode=0
Nov 03 02:16:38.897954 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:38.897478    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt" event={"ID":"50e0635a-54b1-497e-9e2c-66f6961eee54","Type":"ContainerDied","Data":"3741efb97dad30399eb4d28229b7f2d022eb18c1c74bea83ffd216321577d308"}
Nov 03 02:16:38.897954 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:38.897491    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"
Nov 03 02:16:38.897954 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:38.897591    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt" event={"ID":"50e0635a-54b1-497e-9e2c-66f6961eee54","Type":"ContainerDied","Data":"9fa6085572f856823a5d8bded4463baa8b819b23c9ac9fafb01e36f33e557e2d"}
Nov 03 02:16:38.897954 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:38.897611    5005 scope.go:122] "RemoveContainer" containerID="3741efb97dad30399eb4d28229b7f2d022eb18c1c74bea83ffd216321577d308"
Nov 03 02:16:38.910168 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:38.910152    5005 scope.go:122] "RemoveContainer" containerID="3741efb97dad30399eb4d28229b7f2d022eb18c1c74bea83ffd216321577d308"
Nov 03 02:16:38.910470 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:16:38.910454    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3741efb97dad30399eb4d28229b7f2d022eb18c1c74bea83ffd216321577d308\": container with ID starting with 3741efb97dad30399eb4d28229b7f2d022eb18c1c74bea83ffd216321577d308 not found: ID does not exist" containerID="3741efb97dad30399eb4d28229b7f2d022eb18c1c74bea83ffd216321577d308"
Nov 03 02:16:38.910509 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:38.910479    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3741efb97dad30399eb4d28229b7f2d022eb18c1c74bea83ffd216321577d308"} err="failed to get container status \"3741efb97dad30399eb4d28229b7f2d022eb18c1c74bea83ffd216321577d308\": rpc error: code = NotFound desc = could not find container \"3741efb97dad30399eb4d28229b7f2d022eb18c1c74bea83ffd216321577d308\": container with ID starting with 3741efb97dad30399eb4d28229b7f2d022eb18c1c74bea83ffd216321577d308 not found: ID does not exist"
Nov 03 02:16:39.051642 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:39.051608    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"]
Nov 03 02:16:39.098040 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:39.098001    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7f332d-649fcfd989-24plt"]
Nov 03 02:16:39.110282 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:39.110244    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50e0635a-54b1-497e-9e2c-66f6961eee54" path="/var/lib/kubelet/pods/50e0635a-54b1-497e-9e2c-66f6961eee54/volumes"
Nov 03 02:16:48.035233 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:48.035195    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"]
Nov 03 02:16:48.035787 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:48.035394    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr" podUID="55279a3b-4345-47ba-b065-b9ac01da611d" containerName="istio-proxy" containerID="cri-o://effcc350c6ea4e4e0a553135528abb3ff4dd7cea2ec7860158c9a9576bf09d96" gracePeriod=30
Nov 03 02:16:48.081860 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:48.081826    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"]
Nov 03 02:16:48.082060 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:48.082035    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc" podUID="2ac8ea76-22ba-4392-a97c-ca67dc13f427" containerName="istio-proxy" containerID="cri-o://c30c93a191219e82f9044010f380d06b76cfb642fed78a2e6fea9572594097b0" gracePeriod=30
Nov 03 02:16:49.402707 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:49.402665    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"]
Nov 03 02:16:49.403197 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:49.402891    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb" podUID="6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" containerName="istio-proxy" containerID="cri-o://c2b6723120ec8371477eb2312a2dfdf61636388050d269485462419f95b360b5" gracePeriod=30
Nov 03 02:16:53.374563 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.374533    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:16:53.415536 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.415512    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:16:53.500647 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.500607    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-workload-socket\") pod \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") "
Nov 03 02:16:53.500874 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.500657    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-credential-socket\") pod \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") "
Nov 03 02:16:53.500874 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.500727    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-data\") pod \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") "
Nov 03 02:16:53.500874 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.500760    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-podinfo\") pod \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") "
Nov 03 02:16:53.500874 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.500796    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-workload-socket\") pod \"55279a3b-4345-47ba-b065-b9ac01da611d\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") "
Nov 03 02:16:53.500874 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.500822    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55279a3b-4345-47ba-b065-b9ac01da611d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55279a3b-4345-47ba-b065-b9ac01da611d-istiod-ca-cert\") pod \"55279a3b-4345-47ba-b065-b9ac01da611d\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") "
Nov 03 02:16:53.500874 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.500864    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-credential-socket\") pod \"55279a3b-4345-47ba-b065-b9ac01da611d\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") "
Nov 03 02:16:53.501159 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.500917    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-istio-envoy\") pod \"55279a3b-4345-47ba-b065-b9ac01da611d\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") "
Nov 03 02:16:53.501159 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.500972    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55279a3b-4345-47ba-b065-b9ac01da611d-istio-token\" (UniqueName: \"kubernetes.io/projected/55279a3b-4345-47ba-b065-b9ac01da611d-istio-token\") pod \"55279a3b-4345-47ba-b065-b9ac01da611d\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") "
Nov 03 02:16:53.501159 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.501014    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-envoy\") pod \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") "
Nov 03 02:16:53.501159 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.501013    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-credential-socket" pod "2ac8ea76-22ba-4392-a97c-ca67dc13f427" (UID: "2ac8ea76-22ba-4392-a97c-ca67dc13f427"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:53.501159 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.501039    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-workload-socket" pod "2ac8ea76-22ba-4392-a97c-ca67dc13f427" (UID: "2ac8ea76-22ba-4392-a97c-ca67dc13f427"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:53.501159 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.501051    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-data" pod "2ac8ea76-22ba-4392-a97c-ca67dc13f427" (UID: "2ac8ea76-22ba-4392-a97c-ca67dc13f427"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:53.501159 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.501079    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55279a3b-4345-47ba-b065-b9ac01da611d-kube-api-access-d7qhc\" (UniqueName: \"kubernetes.io/projected/55279a3b-4345-47ba-b065-b9ac01da611d-kube-api-access-d7qhc\") pod \"55279a3b-4345-47ba-b065-b9ac01da611d\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") "
Nov 03 02:16:53.501159 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.501113    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-workload-socket" pod "55279a3b-4345-47ba-b065-b9ac01da611d" (UID: "55279a3b-4345-47ba-b065-b9ac01da611d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:53.501159 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.501132    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-workload-certs\") pod \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") "
Nov 03 02:16:53.501159 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.501159    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-token\" (UniqueName: \"kubernetes.io/projected/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-token\") pod \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") "
Nov 03 02:16:53.501680 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.501219    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istiod-ca-cert\") pod \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") "
Nov 03 02:16:53.501680 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.501251    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55279a3b-4345-47ba-b065-b9ac01da611d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55279a3b-4345-47ba-b065-b9ac01da611d-istio-podinfo\") pod \"55279a3b-4345-47ba-b065-b9ac01da611d\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") "
Nov 03 02:16:53.501680 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.501332    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-credential-socket" pod "55279a3b-4345-47ba-b065-b9ac01da611d" (UID: "55279a3b-4345-47ba-b065-b9ac01da611d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:53.501680 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.501335    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-workload-certs\") pod \"55279a3b-4345-47ba-b065-b9ac01da611d\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") "
Nov 03 02:16:53.501680 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.501420    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-istio-data\") pod \"55279a3b-4345-47ba-b065-b9ac01da611d\" (UID: \"55279a3b-4345-47ba-b065-b9ac01da611d\") "
Nov 03 02:16:53.501680 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.501425    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55279a3b-4345-47ba-b065-b9ac01da611d-istiod-ca-cert" pod "55279a3b-4345-47ba-b065-b9ac01da611d" (UID: "55279a3b-4345-47ba-b065-b9ac01da611d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:16:53.501680 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.501539    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-workload-certs" pod "55279a3b-4345-47ba-b065-b9ac01da611d" (UID: "55279a3b-4345-47ba-b065-b9ac01da611d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:53.501680 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.501581    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ac8ea76-22ba-4392-a97c-ca67dc13f427-kube-api-access-hnzbv\" (UniqueName: \"kubernetes.io/projected/2ac8ea76-22ba-4392-a97c-ca67dc13f427-kube-api-access-hnzbv\") pod \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\" (UID: \"2ac8ea76-22ba-4392-a97c-ca67dc13f427\") "
Nov 03 02:16:53.502131 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.501924    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-istio-data" pod "55279a3b-4345-47ba-b065-b9ac01da611d" (UID: "55279a3b-4345-47ba-b065-b9ac01da611d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:53.502131 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.501957    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istiod-ca-cert" pod "2ac8ea76-22ba-4392-a97c-ca67dc13f427" (UID: "2ac8ea76-22ba-4392-a97c-ca67dc13f427"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:16:53.502246 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.502189    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-workload-certs" pod "2ac8ea76-22ba-4392-a97c-ca67dc13f427" (UID: "2ac8ea76-22ba-4392-a97c-ca67dc13f427"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:53.502305 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.502278    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:53.502366 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.502315    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:53.502366 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.502332    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:53.502366 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.502346    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:53.502366 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.502361    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:53.502574 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.502376    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:53.502574 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.502391    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55279a3b-4345-47ba-b065-b9ac01da611d-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:53.502574 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.502404    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:53.502574 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.502419    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:53.503866 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.503834    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-istio-envoy" pod "55279a3b-4345-47ba-b065-b9ac01da611d" (UID: "55279a3b-4345-47ba-b065-b9ac01da611d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:53.503977 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.503951    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-token" pod "2ac8ea76-22ba-4392-a97c-ca67dc13f427" (UID: "2ac8ea76-22ba-4392-a97c-ca67dc13f427"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:16:53.504250 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.504224    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55279a3b-4345-47ba-b065-b9ac01da611d-istio-token" pod "55279a3b-4345-47ba-b065-b9ac01da611d" (UID: "55279a3b-4345-47ba-b065-b9ac01da611d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:16:53.504371 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.504285    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac8ea76-22ba-4392-a97c-ca67dc13f427-kube-api-access-hnzbv" pod "2ac8ea76-22ba-4392-a97c-ca67dc13f427" (UID: "2ac8ea76-22ba-4392-a97c-ca67dc13f427"). InnerVolumeSpecName "kube-api-access-hnzbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:16:53.504371 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.504344    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-envoy" pod "2ac8ea76-22ba-4392-a97c-ca67dc13f427" (UID: "2ac8ea76-22ba-4392-a97c-ca67dc13f427"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:53.504487 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.504425    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-podinfo" pod "2ac8ea76-22ba-4392-a97c-ca67dc13f427" (UID: "2ac8ea76-22ba-4392-a97c-ca67dc13f427"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:16:53.504547 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.504531    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55279a3b-4345-47ba-b065-b9ac01da611d-istio-podinfo" pod "55279a3b-4345-47ba-b065-b9ac01da611d" (UID: "55279a3b-4345-47ba-b065-b9ac01da611d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:16:53.505374 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.505353    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55279a3b-4345-47ba-b065-b9ac01da611d-kube-api-access-d7qhc" pod "55279a3b-4345-47ba-b065-b9ac01da611d" (UID: "55279a3b-4345-47ba-b065-b9ac01da611d"). InnerVolumeSpecName "kube-api-access-d7qhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:16:53.603751 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.603638    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:53.603751 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.603671    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55279a3b-4345-47ba-b065-b9ac01da611d-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:53.603751 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.603681    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55279a3b-4345-47ba-b065-b9ac01da611d-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:53.603751 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.603703    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:53.603751 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.603713    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7qhc\" (UniqueName: \"kubernetes.io/projected/55279a3b-4345-47ba-b065-b9ac01da611d-kube-api-access-d7qhc\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:53.603751 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.603722    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ac8ea76-22ba-4392-a97c-ca67dc13f427-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:53.603751 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.603736    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ac8ea76-22ba-4392-a97c-ca67dc13f427-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:53.603751 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.603750    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55279a3b-4345-47ba-b065-b9ac01da611d-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:53.603751 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:53.603762    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnzbv\" (UniqueName: \"kubernetes.io/projected/2ac8ea76-22ba-4392-a97c-ca67dc13f427-kube-api-access-hnzbv\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:54.014869 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.014835    5005 generic.go:356] "Generic (PLEG): container finished" podID="2ac8ea76-22ba-4392-a97c-ca67dc13f427" containerID="c30c93a191219e82f9044010f380d06b76cfb642fed78a2e6fea9572594097b0" exitCode=0
Nov 03 02:16:54.015089 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.014888    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"
Nov 03 02:16:54.015089 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.014913    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc" event={"ID":"2ac8ea76-22ba-4392-a97c-ca67dc13f427","Type":"ContainerDied","Data":"c30c93a191219e82f9044010f380d06b76cfb642fed78a2e6fea9572594097b0"}
Nov 03 02:16:54.015089 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.014989    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc" event={"ID":"2ac8ea76-22ba-4392-a97c-ca67dc13f427","Type":"ContainerDied","Data":"d42dcea4f8856e28f772a560927b81f4610a17736504a5af88f805c208a9e771"}
Nov 03 02:16:54.015089 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.015020    5005 scope.go:122] "RemoveContainer" containerID="c30c93a191219e82f9044010f380d06b76cfb642fed78a2e6fea9572594097b0"
Nov 03 02:16:54.016988 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.016961    5005 generic.go:356] "Generic (PLEG): container finished" podID="55279a3b-4345-47ba-b065-b9ac01da611d" containerID="effcc350c6ea4e4e0a553135528abb3ff4dd7cea2ec7860158c9a9576bf09d96" exitCode=0
Nov 03 02:16:54.017116 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.017020    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"
Nov 03 02:16:54.017116 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.017059    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr" event={"ID":"55279a3b-4345-47ba-b065-b9ac01da611d","Type":"ContainerDied","Data":"effcc350c6ea4e4e0a553135528abb3ff4dd7cea2ec7860158c9a9576bf09d96"}
Nov 03 02:16:54.017239 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.017120    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr" event={"ID":"55279a3b-4345-47ba-b065-b9ac01da611d","Type":"ContainerDied","Data":"ca400936629a4813505b68a5fe5cf411d435ab76f1c3fc5ac552795d38c8ea37"}
Nov 03 02:16:54.029999 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.029232    5005 scope.go:122] "RemoveContainer" containerID="c30c93a191219e82f9044010f380d06b76cfb642fed78a2e6fea9572594097b0"
Nov 03 02:16:54.029999 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:16:54.029588    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c30c93a191219e82f9044010f380d06b76cfb642fed78a2e6fea9572594097b0\": container with ID starting with c30c93a191219e82f9044010f380d06b76cfb642fed78a2e6fea9572594097b0 not found: ID does not exist" containerID="c30c93a191219e82f9044010f380d06b76cfb642fed78a2e6fea9572594097b0"
Nov 03 02:16:54.029999 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.029639    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30c93a191219e82f9044010f380d06b76cfb642fed78a2e6fea9572594097b0"} err="failed to get container status \"c30c93a191219e82f9044010f380d06b76cfb642fed78a2e6fea9572594097b0\": rpc error: code = NotFound desc = could not find container \"c30c93a191219e82f9044010f380d06b76cfb642fed78a2e6fea9572594097b0\": container with ID starting with c30c93a191219e82f9044010f380d06b76cfb642fed78a2e6fea9572594097b0 not found: ID does not exist"
Nov 03 02:16:54.029999 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.029666    5005 scope.go:122] "RemoveContainer" containerID="effcc350c6ea4e4e0a553135528abb3ff4dd7cea2ec7860158c9a9576bf09d96"
Nov 03 02:16:54.040650 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.040623    5005 scope.go:122] "RemoveContainer" containerID="effcc350c6ea4e4e0a553135528abb3ff4dd7cea2ec7860158c9a9576bf09d96"
Nov 03 02:16:54.040939 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:16:54.040921    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"effcc350c6ea4e4e0a553135528abb3ff4dd7cea2ec7860158c9a9576bf09d96\": container with ID starting with effcc350c6ea4e4e0a553135528abb3ff4dd7cea2ec7860158c9a9576bf09d96 not found: ID does not exist" containerID="effcc350c6ea4e4e0a553135528abb3ff4dd7cea2ec7860158c9a9576bf09d96"
Nov 03 02:16:54.040992 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.040948    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"effcc350c6ea4e4e0a553135528abb3ff4dd7cea2ec7860158c9a9576bf09d96"} err="failed to get container status \"effcc350c6ea4e4e0a553135528abb3ff4dd7cea2ec7860158c9a9576bf09d96\": rpc error: code = NotFound desc = could not find container \"effcc350c6ea4e4e0a553135528abb3ff4dd7cea2ec7860158c9a9576bf09d96\": container with ID starting with effcc350c6ea4e4e0a553135528abb3ff4dd7cea2ec7860158c9a9576bf09d96 not found: ID does not exist"
Nov 03 02:16:54.172405 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.172355    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"]
Nov 03 02:16:54.216078 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.216050    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-9z9pr"]
Nov 03 02:16:54.321137 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.321044    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"]
Nov 03 02:16:54.324873 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.324846    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jmxsc"]
Nov 03 02:16:54.715614 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.715588    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:16:54.821137 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.821103    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-token\" (UniqueName: \"kubernetes.io/projected/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-token\") pod \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") "
Nov 03 02:16:54.821343 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.821152    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-credential-socket\") pod \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") "
Nov 03 02:16:54.821343 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.821190    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-kube-api-access-44sxb\" (UniqueName: \"kubernetes.io/projected/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-kube-api-access-44sxb\") pod \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") "
Nov 03 02:16:54.821343 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.821228    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istiod-ca-cert\") pod \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") "
Nov 03 02:16:54.821343 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.821261    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-envoy\") pod \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") "
Nov 03 02:16:54.821577 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.821354    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-workload-certs\") pod \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") "
Nov 03 02:16:54.821577 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.821419    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-workload-socket\") pod \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") "
Nov 03 02:16:54.821577 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.821506    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-podinfo\") pod \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") "
Nov 03 02:16:54.821577 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.821505    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-credential-socket" pod "6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" (UID: "6f80420e-9ed1-46ea-9fb1-c5489c72ed4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:54.821828 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.821673    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-data\") pod \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\" (UID: \"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c\") "
Nov 03 02:16:54.821828 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.821759    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-workload-certs" pod "6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" (UID: "6f80420e-9ed1-46ea-9fb1-c5489c72ed4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:54.821828 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.821766    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-workload-socket" pod "6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" (UID: "6f80420e-9ed1-46ea-9fb1-c5489c72ed4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:54.821985 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.821937    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istiod-ca-cert" pod "6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" (UID: "6f80420e-9ed1-46ea-9fb1-c5489c72ed4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:16:54.822088 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.822067    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-data" pod "6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" (UID: "6f80420e-9ed1-46ea-9fb1-c5489c72ed4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:54.822382 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.822359    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:54.822480 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.822383    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:54.822480 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.822415    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:54.822480 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.822430    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:54.822480 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.822443    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:54.823654 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.823629    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-token" pod "6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" (UID: "6f80420e-9ed1-46ea-9fb1-c5489c72ed4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:16:54.823778 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.823678    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-kube-api-access-44sxb" pod "6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" (UID: "6f80420e-9ed1-46ea-9fb1-c5489c72ed4c"). InnerVolumeSpecName "kube-api-access-44sxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:16:54.823965 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.823944    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-podinfo" pod "6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" (UID: "6f80420e-9ed1-46ea-9fb1-c5489c72ed4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:16:54.824083 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.823959    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-envoy" pod "6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" (UID: "6f80420e-9ed1-46ea-9fb1-c5489c72ed4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:54.923539 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.923504    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:54.923539 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.923534    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:54.923539 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.923544    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44sxb\" (UniqueName: \"kubernetes.io/projected/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-kube-api-access-44sxb\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:54.923829 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:54.923554    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:55.027759 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:55.027723    5005 generic.go:356] "Generic (PLEG): container finished" podID="6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" containerID="c2b6723120ec8371477eb2312a2dfdf61636388050d269485462419f95b360b5" exitCode=0
Nov 03 02:16:55.027932 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:55.027759    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb" event={"ID":"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c","Type":"ContainerDied","Data":"c2b6723120ec8371477eb2312a2dfdf61636388050d269485462419f95b360b5"}
Nov 03 02:16:55.027932 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:55.027786    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"
Nov 03 02:16:55.027932 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:55.027822    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb" event={"ID":"6f80420e-9ed1-46ea-9fb1-c5489c72ed4c","Type":"ContainerDied","Data":"8859fc5a4c518503c23b173c3aca554d22503b00a157a59c1e980406ea2f674b"}
Nov 03 02:16:55.027932 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:55.027846    5005 scope.go:122] "RemoveContainer" containerID="c2b6723120ec8371477eb2312a2dfdf61636388050d269485462419f95b360b5"
Nov 03 02:16:55.044588 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:55.044570    5005 scope.go:122] "RemoveContainer" containerID="c2b6723120ec8371477eb2312a2dfdf61636388050d269485462419f95b360b5"
Nov 03 02:16:55.044889 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:16:55.044863    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b6723120ec8371477eb2312a2dfdf61636388050d269485462419f95b360b5\": container with ID starting with c2b6723120ec8371477eb2312a2dfdf61636388050d269485462419f95b360b5 not found: ID does not exist" containerID="c2b6723120ec8371477eb2312a2dfdf61636388050d269485462419f95b360b5"
Nov 03 02:16:55.044959 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:55.044899    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b6723120ec8371477eb2312a2dfdf61636388050d269485462419f95b360b5"} err="failed to get container status \"c2b6723120ec8371477eb2312a2dfdf61636388050d269485462419f95b360b5\": rpc error: code = NotFound desc = could not find container \"c2b6723120ec8371477eb2312a2dfdf61636388050d269485462419f95b360b5\": container with ID starting with c2b6723120ec8371477eb2312a2dfdf61636388050d269485462419f95b360b5 not found: ID does not exist"
Nov 03 02:16:55.112551 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:55.112467    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ac8ea76-22ba-4392-a97c-ca67dc13f427" path="/var/lib/kubelet/pods/2ac8ea76-22ba-4392-a97c-ca67dc13f427/volumes"
Nov 03 02:16:55.113072 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:55.113053    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55279a3b-4345-47ba-b065-b9ac01da611d" path="/var/lib/kubelet/pods/55279a3b-4345-47ba-b065-b9ac01da611d/volumes"
Nov 03 02:16:55.181632 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:55.181584    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"]
Nov 03 02:16:55.186268 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:55.186223    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-n5nxb"]
Nov 03 02:16:57.111331 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:16:57.111293    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" path="/var/lib/kubelet/pods/6f80420e-9ed1-46ea-9fb1-c5489c72ed4c/volumes"
Nov 03 02:17:00.716587 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:17:00.716538    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:17:08.745075 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:08.745039    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"]
Nov 03 02:17:08.745562 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:08.745317    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd" podUID="3ec530be-e064-476f-b3e3-6cafd2d3d1d4" containerName="maas-api" containerID="cri-o://bfedc3f61fa0d7606e62299fa8a6841887721c9644003a556ead72740069e666" gracePeriod=30
Nov 03 02:17:09.076747 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:09.076718    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"
Nov 03 02:17:09.136159 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:09.136114    5005 generic.go:356] "Generic (PLEG): container finished" podID="3ec530be-e064-476f-b3e3-6cafd2d3d1d4" containerID="bfedc3f61fa0d7606e62299fa8a6841887721c9644003a556ead72740069e666" exitCode=0
Nov 03 02:17:09.136323 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:09.136227    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"
Nov 03 02:17:09.136323 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:09.136247    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd" event={"ID":"3ec530be-e064-476f-b3e3-6cafd2d3d1d4","Type":"ContainerDied","Data":"bfedc3f61fa0d7606e62299fa8a6841887721c9644003a556ead72740069e666"}
Nov 03 02:17:09.136323 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:09.136313    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd" event={"ID":"3ec530be-e064-476f-b3e3-6cafd2d3d1d4","Type":"ContainerDied","Data":"3b8ab9747dc0360e0c03374abe2e65e9824d7feb5ecef57dc582a49fc975e8bb"}
Nov 03 02:17:09.136518 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:09.136345    5005 scope.go:122] "RemoveContainer" containerID="bfedc3f61fa0d7606e62299fa8a6841887721c9644003a556ead72740069e666"
Nov 03 02:17:09.146982 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:09.146963    5005 scope.go:122] "RemoveContainer" containerID="bfedc3f61fa0d7606e62299fa8a6841887721c9644003a556ead72740069e666"
Nov 03 02:17:09.147299 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:17:09.147278    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfedc3f61fa0d7606e62299fa8a6841887721c9644003a556ead72740069e666\": container with ID starting with bfedc3f61fa0d7606e62299fa8a6841887721c9644003a556ead72740069e666 not found: ID does not exist" containerID="bfedc3f61fa0d7606e62299fa8a6841887721c9644003a556ead72740069e666"
Nov 03 02:17:09.147370 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:09.147312    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfedc3f61fa0d7606e62299fa8a6841887721c9644003a556ead72740069e666"} err="failed to get container status \"bfedc3f61fa0d7606e62299fa8a6841887721c9644003a556ead72740069e666\": rpc error: code = NotFound desc = could not find container \"bfedc3f61fa0d7606e62299fa8a6841887721c9644003a556ead72740069e666\": container with ID starting with bfedc3f61fa0d7606e62299fa8a6841887721c9644003a556ead72740069e666 not found: ID does not exist"
Nov 03 02:17:09.178285 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:09.178258    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-metrics-certs\") pod \"3ec530be-e064-476f-b3e3-6cafd2d3d1d4\" (UID: \"3ec530be-e064-476f-b3e3-6cafd2d3d1d4\") "
Nov 03 02:17:09.178285 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:09.178288    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-maas-api-tls\") pod \"3ec530be-e064-476f-b3e3-6cafd2d3d1d4\" (UID: \"3ec530be-e064-476f-b3e3-6cafd2d3d1d4\") "
Nov 03 02:17:09.178510 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:09.178350    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-kube-api-access-qn7xf\" (UniqueName: \"kubernetes.io/projected/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-kube-api-access-qn7xf\") pod \"3ec530be-e064-476f-b3e3-6cafd2d3d1d4\" (UID: \"3ec530be-e064-476f-b3e3-6cafd2d3d1d4\") "
Nov 03 02:17:09.178510 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:09.178372    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-openshift-service-ca\") pod \"3ec530be-e064-476f-b3e3-6cafd2d3d1d4\" (UID: \"3ec530be-e064-476f-b3e3-6cafd2d3d1d4\") "
Nov 03 02:17:09.178816 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:09.178779    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-openshift-service-ca" pod "3ec530be-e064-476f-b3e3-6cafd2d3d1d4" (UID: "3ec530be-e064-476f-b3e3-6cafd2d3d1d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:17:09.179331 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:09.179304    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:09.180622 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:09.180590    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-maas-api-tls" pod "3ec530be-e064-476f-b3e3-6cafd2d3d1d4" (UID: "3ec530be-e064-476f-b3e3-6cafd2d3d1d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:17:09.180622 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:09.180593    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-kube-api-access-qn7xf" pod "3ec530be-e064-476f-b3e3-6cafd2d3d1d4" (UID: "3ec530be-e064-476f-b3e3-6cafd2d3d1d4"). InnerVolumeSpecName "kube-api-access-qn7xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:17:09.180824 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:09.180659    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-metrics-certs" pod "3ec530be-e064-476f-b3e3-6cafd2d3d1d4" (UID: "3ec530be-e064-476f-b3e3-6cafd2d3d1d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:17:09.280574 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:09.280536    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:09.280574 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:09.280570    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:09.280574 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:09.280582    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn7xf\" (UniqueName: \"kubernetes.io/projected/3ec530be-e064-476f-b3e3-6cafd2d3d1d4-kube-api-access-qn7xf\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:09.551459 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:09.551424    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"]
Nov 03 02:17:09.556047 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:09.556017    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-701ba30f-7869c47c95-l9dsd"]
Nov 03 02:17:11.111149 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:11.111110    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ec530be-e064-476f-b3e3-6cafd2d3d1d4" path="/var/lib/kubelet/pods/3ec530be-e064-476f-b3e3-6cafd2d3d1d4/volumes"
Nov 03 02:17:30.312295 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:30.312216    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"]
Nov 03 02:17:30.312862 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:30.312472    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm" podUID="c4637951-2a4e-4796-8b07-179acbdc0cfb" containerName="maas-api" containerID="cri-o://fc5c2d6048b147da22eb51f174a5486c3d5cc374b3f25b75c423bf1044088e30" gracePeriod=30
Nov 03 02:17:30.613293 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:30.613263    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"
Nov 03 02:17:30.741735 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:30.724901    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4637951-2a4e-4796-8b07-179acbdc0cfb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4637951-2a4e-4796-8b07-179acbdc0cfb-maas-api-tls\") pod \"c4637951-2a4e-4796-8b07-179acbdc0cfb\" (UID: \"c4637951-2a4e-4796-8b07-179acbdc0cfb\") "
Nov 03 02:17:30.741735 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:17:30.725164    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:17:30.741735 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:30.735570    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4637951-2a4e-4796-8b07-179acbdc0cfb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4637951-2a4e-4796-8b07-179acbdc0cfb-metrics-certs\") pod \"c4637951-2a4e-4796-8b07-179acbdc0cfb\" (UID: \"c4637951-2a4e-4796-8b07-179acbdc0cfb\") "
Nov 03 02:17:30.741735 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:30.735824    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4637951-2a4e-4796-8b07-179acbdc0cfb-kube-api-access-gmdcr\" (UniqueName: \"kubernetes.io/projected/c4637951-2a4e-4796-8b07-179acbdc0cfb-kube-api-access-gmdcr\") pod \"c4637951-2a4e-4796-8b07-179acbdc0cfb\" (UID: \"c4637951-2a4e-4796-8b07-179acbdc0cfb\") "
Nov 03 02:17:30.741735 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:30.735870    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4637951-2a4e-4796-8b07-179acbdc0cfb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4637951-2a4e-4796-8b07-179acbdc0cfb-openshift-service-ca\") pod \"c4637951-2a4e-4796-8b07-179acbdc0cfb\" (UID: \"c4637951-2a4e-4796-8b07-179acbdc0cfb\") "
Nov 03 02:17:30.741735 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:30.736911    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4637951-2a4e-4796-8b07-179acbdc0cfb-openshift-service-ca" pod "c4637951-2a4e-4796-8b07-179acbdc0cfb" (UID: "c4637951-2a4e-4796-8b07-179acbdc0cfb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:17:30.742182 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:30.742138    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4637951-2a4e-4796-8b07-179acbdc0cfb-metrics-certs" pod "c4637951-2a4e-4796-8b07-179acbdc0cfb" (UID: "c4637951-2a4e-4796-8b07-179acbdc0cfb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:17:30.742182 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:30.742158    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4637951-2a4e-4796-8b07-179acbdc0cfb-maas-api-tls" pod "c4637951-2a4e-4796-8b07-179acbdc0cfb" (UID: "c4637951-2a4e-4796-8b07-179acbdc0cfb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:17:30.742319 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:30.742181    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4637951-2a4e-4796-8b07-179acbdc0cfb-kube-api-access-gmdcr" pod "c4637951-2a4e-4796-8b07-179acbdc0cfb" (UID: "c4637951-2a4e-4796-8b07-179acbdc0cfb"). InnerVolumeSpecName "kube-api-access-gmdcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:17:30.842820 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:30.839082    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4637951-2a4e-4796-8b07-179acbdc0cfb-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:30.842820 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:30.839123    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmdcr\" (UniqueName: \"kubernetes.io/projected/c4637951-2a4e-4796-8b07-179acbdc0cfb-kube-api-access-gmdcr\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:30.842820 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:30.839143    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4637951-2a4e-4796-8b07-179acbdc0cfb-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:30.842820 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:30.839161    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4637951-2a4e-4796-8b07-179acbdc0cfb-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:31.307477 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:31.307442    5005 generic.go:356] "Generic (PLEG): container finished" podID="c4637951-2a4e-4796-8b07-179acbdc0cfb" containerID="fc5c2d6048b147da22eb51f174a5486c3d5cc374b3f25b75c423bf1044088e30" exitCode=0
Nov 03 02:17:31.307658 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:31.307501    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"
Nov 03 02:17:31.307658 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:31.307527    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm" event={"ID":"c4637951-2a4e-4796-8b07-179acbdc0cfb","Type":"ContainerDied","Data":"fc5c2d6048b147da22eb51f174a5486c3d5cc374b3f25b75c423bf1044088e30"}
Nov 03 02:17:31.307658 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:31.307596    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm" event={"ID":"c4637951-2a4e-4796-8b07-179acbdc0cfb","Type":"ContainerDied","Data":"7c04174f37f9d795f2f95033312e3596dd571e50da36f7f785bb0e854b680b21"}
Nov 03 02:17:31.307658 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:31.307621    5005 scope.go:122] "RemoveContainer" containerID="fc5c2d6048b147da22eb51f174a5486c3d5cc374b3f25b75c423bf1044088e30"
Nov 03 02:17:31.323927 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:31.322422    5005 scope.go:122] "RemoveContainer" containerID="fc5c2d6048b147da22eb51f174a5486c3d5cc374b3f25b75c423bf1044088e30"
Nov 03 02:17:31.332059 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:17:31.332026    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc5c2d6048b147da22eb51f174a5486c3d5cc374b3f25b75c423bf1044088e30\": container with ID starting with fc5c2d6048b147da22eb51f174a5486c3d5cc374b3f25b75c423bf1044088e30 not found: ID does not exist" containerID="fc5c2d6048b147da22eb51f174a5486c3d5cc374b3f25b75c423bf1044088e30"
Nov 03 02:17:31.332291 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:31.332262    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc5c2d6048b147da22eb51f174a5486c3d5cc374b3f25b75c423bf1044088e30"} err="failed to get container status \"fc5c2d6048b147da22eb51f174a5486c3d5cc374b3f25b75c423bf1044088e30\": rpc error: code = NotFound desc = could not find container \"fc5c2d6048b147da22eb51f174a5486c3d5cc374b3f25b75c423bf1044088e30\": container with ID starting with fc5c2d6048b147da22eb51f174a5486c3d5cc374b3f25b75c423bf1044088e30 not found: ID does not exist"
Nov 03 02:17:31.465133 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:31.465033    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"]
Nov 03 02:17:31.469806 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:31.469780    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a772c3-5ffb58bdb7-c6cmm"]
Nov 03 02:17:33.112707 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:33.112656    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4637951-2a4e-4796-8b07-179acbdc0cfb" path="/var/lib/kubelet/pods/c4637951-2a4e-4796-8b07-179acbdc0cfb/volumes"
Nov 03 02:17:35.348252 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:35.348215    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"]
Nov 03 02:17:35.348772 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:35.348504    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk" podUID="28e48d72-ce80-4a8e-ada5-c5f7f3cda02f" containerName="istio-proxy" containerID="cri-o://5fe083e68b42ea772960b4ccb1eb98acc68d1a563bca20523e38594c7b967186" gracePeriod=30
Nov 03 02:17:36.712672 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:36.712636    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"]
Nov 03 02:17:36.713135 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:36.712846    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7" podUID="b02bc9ee-d26b-48b2-b2ce-06784e778bce" containerName="maas-api" containerID="cri-o://fb89e970b0de336b2b57d20e9b0053cf123e3f1406057ec3f0b13ebc9477a007" gracePeriod=30
Nov 03 02:17:37.009545 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:37.009217    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"
Nov 03 02:17:37.075281 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:37.075239    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b02bc9ee-d26b-48b2-b2ce-06784e778bce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02bc9ee-d26b-48b2-b2ce-06784e778bce-openshift-service-ca\") pod \"b02bc9ee-d26b-48b2-b2ce-06784e778bce\" (UID: \"b02bc9ee-d26b-48b2-b2ce-06784e778bce\") "
Nov 03 02:17:37.075466 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:37.075389    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b02bc9ee-d26b-48b2-b2ce-06784e778bce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b02bc9ee-d26b-48b2-b2ce-06784e778bce-metrics-certs\") pod \"b02bc9ee-d26b-48b2-b2ce-06784e778bce\" (UID: \"b02bc9ee-d26b-48b2-b2ce-06784e778bce\") "
Nov 03 02:17:37.075466 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:37.075430    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b02bc9ee-d26b-48b2-b2ce-06784e778bce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02bc9ee-d26b-48b2-b2ce-06784e778bce-maas-api-tls\") pod \"b02bc9ee-d26b-48b2-b2ce-06784e778bce\" (UID: \"b02bc9ee-d26b-48b2-b2ce-06784e778bce\") "
Nov 03 02:17:37.075595 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:37.075506    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02bc9ee-d26b-48b2-b2ce-06784e778bce-kube-api-access-shv2g\" (UniqueName: \"kubernetes.io/projected/b02bc9ee-d26b-48b2-b2ce-06784e778bce-kube-api-access-shv2g\") pod \"b02bc9ee-d26b-48b2-b2ce-06784e778bce\" (UID: \"b02bc9ee-d26b-48b2-b2ce-06784e778bce\") "
Nov 03 02:17:37.075751 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:37.075721    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02bc9ee-d26b-48b2-b2ce-06784e778bce-openshift-service-ca" pod "b02bc9ee-d26b-48b2-b2ce-06784e778bce" (UID: "b02bc9ee-d26b-48b2-b2ce-06784e778bce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:17:37.076065 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:37.076044    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02bc9ee-d26b-48b2-b2ce-06784e778bce-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:37.078138 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:37.078112    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b02bc9ee-d26b-48b2-b2ce-06784e778bce-maas-api-tls" pod "b02bc9ee-d26b-48b2-b2ce-06784e778bce" (UID: "b02bc9ee-d26b-48b2-b2ce-06784e778bce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:17:37.078242 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:37.078133    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02bc9ee-d26b-48b2-b2ce-06784e778bce-kube-api-access-shv2g" pod "b02bc9ee-d26b-48b2-b2ce-06784e778bce" (UID: "b02bc9ee-d26b-48b2-b2ce-06784e778bce"). InnerVolumeSpecName "kube-api-access-shv2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:17:37.078242 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:37.078144    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b02bc9ee-d26b-48b2-b2ce-06784e778bce-metrics-certs" pod "b02bc9ee-d26b-48b2-b2ce-06784e778bce" (UID: "b02bc9ee-d26b-48b2-b2ce-06784e778bce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:17:37.177827 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:37.177792    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b02bc9ee-d26b-48b2-b2ce-06784e778bce-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:37.177827 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:37.177829    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02bc9ee-d26b-48b2-b2ce-06784e778bce-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:37.178089 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:37.177844    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shv2g\" (UniqueName: \"kubernetes.io/projected/b02bc9ee-d26b-48b2-b2ce-06784e778bce-kube-api-access-shv2g\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:37.360544 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:37.360509    5005 generic.go:356] "Generic (PLEG): container finished" podID="b02bc9ee-d26b-48b2-b2ce-06784e778bce" containerID="fb89e970b0de336b2b57d20e9b0053cf123e3f1406057ec3f0b13ebc9477a007" exitCode=0
Nov 03 02:17:37.361099 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:37.360585    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"
Nov 03 02:17:37.361099 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:37.360802    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7" event={"ID":"b02bc9ee-d26b-48b2-b2ce-06784e778bce","Type":"ContainerDied","Data":"fb89e970b0de336b2b57d20e9b0053cf123e3f1406057ec3f0b13ebc9477a007"}
Nov 03 02:17:37.361099 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:37.360860    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7" event={"ID":"b02bc9ee-d26b-48b2-b2ce-06784e778bce","Type":"ContainerDied","Data":"372d6bab490cc25131ff20baed61ad3ee98e6d6ecedef2a600439e5dd2b1ccd1"}
Nov 03 02:17:37.361099 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:37.360894    5005 scope.go:122] "RemoveContainer" containerID="fb89e970b0de336b2b57d20e9b0053cf123e3f1406057ec3f0b13ebc9477a007"
Nov 03 02:17:37.371909 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:37.371889    5005 scope.go:122] "RemoveContainer" containerID="fb89e970b0de336b2b57d20e9b0053cf123e3f1406057ec3f0b13ebc9477a007"
Nov 03 02:17:37.372237 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:17:37.372217    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb89e970b0de336b2b57d20e9b0053cf123e3f1406057ec3f0b13ebc9477a007\": container with ID starting with fb89e970b0de336b2b57d20e9b0053cf123e3f1406057ec3f0b13ebc9477a007 not found: ID does not exist" containerID="fb89e970b0de336b2b57d20e9b0053cf123e3f1406057ec3f0b13ebc9477a007"
Nov 03 02:17:37.372299 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:37.372249    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb89e970b0de336b2b57d20e9b0053cf123e3f1406057ec3f0b13ebc9477a007"} err="failed to get container status \"fb89e970b0de336b2b57d20e9b0053cf123e3f1406057ec3f0b13ebc9477a007\": rpc error: code = NotFound desc = could not find container \"fb89e970b0de336b2b57d20e9b0053cf123e3f1406057ec3f0b13ebc9477a007\": container with ID starting with fb89e970b0de336b2b57d20e9b0053cf123e3f1406057ec3f0b13ebc9477a007 not found: ID does not exist"
Nov 03 02:17:37.516029 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:37.515973    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"]
Nov 03 02:17:37.522504 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:37.522475    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c617de-76d6557dbf-8hls7"]
Nov 03 02:17:39.110057 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:39.110023    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b02bc9ee-d26b-48b2-b2ce-06784e778bce" path="/var/lib/kubelet/pods/b02bc9ee-d26b-48b2-b2ce-06784e778bce/volumes"
Nov 03 02:17:40.614335 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.614275    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:17:40.728499 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.728466    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-workload-certs\") pod \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") "
Nov 03 02:17:40.728750 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.728516    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-credential-socket\") pod \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") "
Nov 03 02:17:40.728750 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.728542    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-token\" (UniqueName: \"kubernetes.io/projected/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-token\") pod \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") "
Nov 03 02:17:40.728750 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.728583    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-kube-api-access-zbv7x\" (UniqueName: \"kubernetes.io/projected/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-kube-api-access-zbv7x\") pod \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") "
Nov 03 02:17:40.728750 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.728618    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-podinfo\") pod \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") "
Nov 03 02:17:40.728750 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.728660    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-workload-socket\") pod \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") "
Nov 03 02:17:40.728750 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.728723    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-data\") pod \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") "
Nov 03 02:17:40.728750 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.728745    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istiod-ca-cert\") pod \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") "
Nov 03 02:17:40.729130 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.728800    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-envoy\") pod \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\" (UID: \"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f\") "
Nov 03 02:17:40.729130 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.728797    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-workload-certs" pod "28e48d72-ce80-4a8e-ada5-c5f7f3cda02f" (UID: "28e48d72-ce80-4a8e-ada5-c5f7f3cda02f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:17:40.729130 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.728977    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-data" pod "28e48d72-ce80-4a8e-ada5-c5f7f3cda02f" (UID: "28e48d72-ce80-4a8e-ada5-c5f7f3cda02f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:17:40.729130 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.729011    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-workload-socket" pod "28e48d72-ce80-4a8e-ada5-c5f7f3cda02f" (UID: "28e48d72-ce80-4a8e-ada5-c5f7f3cda02f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:17:40.729338 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.729185    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istiod-ca-cert" pod "28e48d72-ce80-4a8e-ada5-c5f7f3cda02f" (UID: "28e48d72-ce80-4a8e-ada5-c5f7f3cda02f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:17:40.729455 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.729401    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-credential-socket" pod "28e48d72-ce80-4a8e-ada5-c5f7f3cda02f" (UID: "28e48d72-ce80-4a8e-ada5-c5f7f3cda02f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:17:40.729611 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.729592    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:40.729679 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.729616    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:40.729679 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.729632    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:40.729679 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.729647    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:40.729679 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.729662    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:40.731348 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.731325    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-podinfo" pod "28e48d72-ce80-4a8e-ada5-c5f7f3cda02f" (UID: "28e48d72-ce80-4a8e-ada5-c5f7f3cda02f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:17:40.731541 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.731515    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-kube-api-access-zbv7x" pod "28e48d72-ce80-4a8e-ada5-c5f7f3cda02f" (UID: "28e48d72-ce80-4a8e-ada5-c5f7f3cda02f"). InnerVolumeSpecName "kube-api-access-zbv7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:17:40.731632 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.731571    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-envoy" pod "28e48d72-ce80-4a8e-ada5-c5f7f3cda02f" (UID: "28e48d72-ce80-4a8e-ada5-c5f7f3cda02f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:17:40.731819 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.731796    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-token" pod "28e48d72-ce80-4a8e-ada5-c5f7f3cda02f" (UID: "28e48d72-ce80-4a8e-ada5-c5f7f3cda02f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:17:40.831454 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.831420    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbv7x\" (UniqueName: \"kubernetes.io/projected/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-kube-api-access-zbv7x\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:40.831454 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.831451    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:40.831454 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.831461    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:40.831728 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:40.831470    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:41.392827 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:41.392731    5005 generic.go:356] "Generic (PLEG): container finished" podID="28e48d72-ce80-4a8e-ada5-c5f7f3cda02f" containerID="5fe083e68b42ea772960b4ccb1eb98acc68d1a563bca20523e38594c7b967186" exitCode=0
Nov 03 02:17:41.392827 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:41.392791    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"
Nov 03 02:17:41.392827 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:41.392811    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk" event={"ID":"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f","Type":"ContainerDied","Data":"5fe083e68b42ea772960b4ccb1eb98acc68d1a563bca20523e38594c7b967186"}
Nov 03 02:17:41.393081 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:41.392873    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk" event={"ID":"28e48d72-ce80-4a8e-ada5-c5f7f3cda02f","Type":"ContainerDied","Data":"6773200885dfb4170a019544e8ce091f206dca5645c2b1e6a1f2a9a3e2259be7"}
Nov 03 02:17:41.393081 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:41.392896    5005 scope.go:122] "RemoveContainer" containerID="5fe083e68b42ea772960b4ccb1eb98acc68d1a563bca20523e38594c7b967186"
Nov 03 02:17:41.402777 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:41.402755    5005 scope.go:122] "RemoveContainer" containerID="5fe083e68b42ea772960b4ccb1eb98acc68d1a563bca20523e38594c7b967186"
Nov 03 02:17:41.403067 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:17:41.403046    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fe083e68b42ea772960b4ccb1eb98acc68d1a563bca20523e38594c7b967186\": container with ID starting with 5fe083e68b42ea772960b4ccb1eb98acc68d1a563bca20523e38594c7b967186 not found: ID does not exist" containerID="5fe083e68b42ea772960b4ccb1eb98acc68d1a563bca20523e38594c7b967186"
Nov 03 02:17:41.403128 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:41.403076    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe083e68b42ea772960b4ccb1eb98acc68d1a563bca20523e38594c7b967186"} err="failed to get container status \"5fe083e68b42ea772960b4ccb1eb98acc68d1a563bca20523e38594c7b967186\": rpc error: code = NotFound desc = could not find container \"5fe083e68b42ea772960b4ccb1eb98acc68d1a563bca20523e38594c7b967186\": container with ID starting with 5fe083e68b42ea772960b4ccb1eb98acc68d1a563bca20523e38594c7b967186 not found: ID does not exist"
Nov 03 02:17:41.548648 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:41.548609    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"]
Nov 03 02:17:41.552185 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:41.552144    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-7pqlk"]
Nov 03 02:17:43.135165 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:17:43.135119    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28e48d72-ce80-4a8e-ada5-c5f7f3cda02f" path="/var/lib/kubelet/pods/28e48d72-ce80-4a8e-ada5-c5f7f3cda02f/volumes"
Nov 03 02:18:00.717156 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:18:00.717116    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:18:27.831799 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:27.831763    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"]
Nov 03 02:18:27.832432 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:27.831980    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w" podUID="1f8fac4d-a231-421f-993e-981c21ca4e81" containerName="maas-api" containerID="cri-o://9cf45661c6134ef66262196379a2bf457d0112205b3121d2dacf7bc1f83f28fe" gracePeriod=30
Nov 03 02:18:28.147265 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:28.147238    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"
Nov 03 02:18:28.204184 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:28.204150    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f8fac4d-a231-421f-993e-981c21ca4e81-kube-api-access-rh7fr\" (UniqueName: \"kubernetes.io/projected/1f8fac4d-a231-421f-993e-981c21ca4e81-kube-api-access-rh7fr\") pod \"1f8fac4d-a231-421f-993e-981c21ca4e81\" (UID: \"1f8fac4d-a231-421f-993e-981c21ca4e81\") "
Nov 03 02:18:28.204383 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:28.204257    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f8fac4d-a231-421f-993e-981c21ca4e81-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f8fac4d-a231-421f-993e-981c21ca4e81-metrics-certs\") pod \"1f8fac4d-a231-421f-993e-981c21ca4e81\" (UID: \"1f8fac4d-a231-421f-993e-981c21ca4e81\") "
Nov 03 02:18:28.204383 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:28.204356    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f8fac4d-a231-421f-993e-981c21ca4e81-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f8fac4d-a231-421f-993e-981c21ca4e81-maas-api-tls\") pod \"1f8fac4d-a231-421f-993e-981c21ca4e81\" (UID: \"1f8fac4d-a231-421f-993e-981c21ca4e81\") "
Nov 03 02:18:28.204508 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:28.204429    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f8fac4d-a231-421f-993e-981c21ca4e81-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8fac4d-a231-421f-993e-981c21ca4e81-openshift-service-ca\") pod \"1f8fac4d-a231-421f-993e-981c21ca4e81\" (UID: \"1f8fac4d-a231-421f-993e-981c21ca4e81\") "
Nov 03 02:18:28.204920 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:28.204869    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f8fac4d-a231-421f-993e-981c21ca4e81-openshift-service-ca" pod "1f8fac4d-a231-421f-993e-981c21ca4e81" (UID: "1f8fac4d-a231-421f-993e-981c21ca4e81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:18:28.205349 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:28.205126    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8fac4d-a231-421f-993e-981c21ca4e81-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:18:28.206738 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:28.206703    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f8fac4d-a231-421f-993e-981c21ca4e81-maas-api-tls" pod "1f8fac4d-a231-421f-993e-981c21ca4e81" (UID: "1f8fac4d-a231-421f-993e-981c21ca4e81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:18:28.206738 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:28.206726    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f8fac4d-a231-421f-993e-981c21ca4e81-metrics-certs" pod "1f8fac4d-a231-421f-993e-981c21ca4e81" (UID: "1f8fac4d-a231-421f-993e-981c21ca4e81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:18:28.206889 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:28.206737    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f8fac4d-a231-421f-993e-981c21ca4e81-kube-api-access-rh7fr" pod "1f8fac4d-a231-421f-993e-981c21ca4e81" (UID: "1f8fac4d-a231-421f-993e-981c21ca4e81"). InnerVolumeSpecName "kube-api-access-rh7fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:18:28.306208 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:28.306175    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f8fac4d-a231-421f-993e-981c21ca4e81-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:18:28.306385 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:28.306216    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh7fr\" (UniqueName: \"kubernetes.io/projected/1f8fac4d-a231-421f-993e-981c21ca4e81-kube-api-access-rh7fr\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:18:28.306385 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:28.306227    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f8fac4d-a231-421f-993e-981c21ca4e81-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:18:28.727206 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:28.727165    5005 generic.go:356] "Generic (PLEG): container finished" podID="1f8fac4d-a231-421f-993e-981c21ca4e81" containerID="9cf45661c6134ef66262196379a2bf457d0112205b3121d2dacf7bc1f83f28fe" exitCode=0
Nov 03 02:18:28.727428 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:28.727223    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"
Nov 03 02:18:28.727428 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:28.727246    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w" event={"ID":"1f8fac4d-a231-421f-993e-981c21ca4e81","Type":"ContainerDied","Data":"9cf45661c6134ef66262196379a2bf457d0112205b3121d2dacf7bc1f83f28fe"}
Nov 03 02:18:28.727428 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:28.727339    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w" event={"ID":"1f8fac4d-a231-421f-993e-981c21ca4e81","Type":"ContainerDied","Data":"73bbacc78d3efe1ddeff408a722e4d7daabe1e5f0057d2a2f2551af2c917280a"}
Nov 03 02:18:28.727428 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:28.727370    5005 scope.go:122] "RemoveContainer" containerID="9cf45661c6134ef66262196379a2bf457d0112205b3121d2dacf7bc1f83f28fe"
Nov 03 02:18:28.740191 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:28.740164    5005 scope.go:122] "RemoveContainer" containerID="9cf45661c6134ef66262196379a2bf457d0112205b3121d2dacf7bc1f83f28fe"
Nov 03 02:18:28.740562 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:18:28.740532    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cf45661c6134ef66262196379a2bf457d0112205b3121d2dacf7bc1f83f28fe\": container with ID starting with 9cf45661c6134ef66262196379a2bf457d0112205b3121d2dacf7bc1f83f28fe not found: ID does not exist" containerID="9cf45661c6134ef66262196379a2bf457d0112205b3121d2dacf7bc1f83f28fe"
Nov 03 02:18:28.740673 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:28.740573    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cf45661c6134ef66262196379a2bf457d0112205b3121d2dacf7bc1f83f28fe"} err="failed to get container status \"9cf45661c6134ef66262196379a2bf457d0112205b3121d2dacf7bc1f83f28fe\": rpc error: code = NotFound desc = could not find container \"9cf45661c6134ef66262196379a2bf457d0112205b3121d2dacf7bc1f83f28fe\": container with ID starting with 9cf45661c6134ef66262196379a2bf457d0112205b3121d2dacf7bc1f83f28fe not found: ID does not exist"
Nov 03 02:18:28.886331 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:28.886298    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"]
Nov 03 02:18:28.932488 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:28.932448    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-aa82d2-5ccf989469-d6f5w"]
Nov 03 02:18:29.110817 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:29.110785    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f8fac4d-a231-421f-993e-981c21ca4e81" path="/var/lib/kubelet/pods/1f8fac4d-a231-421f-993e-981c21ca4e81/volumes"
Nov 03 02:18:30.719212 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:18:30.719170    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:18:31.002479 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:31.002441    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"]
Nov 03 02:18:31.002828 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:31.002788    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh" podUID="38604d86-a2f2-4b36-b644-64d7ed7444b0" containerName="istio-proxy" containerID="cri-o://0b016549b9272958d4581ce4521cdd11c36de1253d306e13eab9a3a4a6e40681" gracePeriod=30
Nov 03 02:18:36.286074 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.286007    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:18:36.422583 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.422546    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-data\") pod \"38604d86-a2f2-4b36-b644-64d7ed7444b0\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") "
Nov 03 02:18:36.422908 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.422602    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-token\" (UniqueName: \"kubernetes.io/projected/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-token\") pod \"38604d86-a2f2-4b36-b644-64d7ed7444b0\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") "
Nov 03 02:18:36.422908 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.422758    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-envoy\") pod \"38604d86-a2f2-4b36-b644-64d7ed7444b0\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") "
Nov 03 02:18:36.422908 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.422812    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-workload-socket\") pod \"38604d86-a2f2-4b36-b644-64d7ed7444b0\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") "
Nov 03 02:18:36.422908 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.422844    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-workload-certs\") pod \"38604d86-a2f2-4b36-b644-64d7ed7444b0\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") "
Nov 03 02:18:36.422908 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.422901    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38604d86-a2f2-4b36-b644-64d7ed7444b0-kube-api-access-sccp7\" (UniqueName: \"kubernetes.io/projected/38604d86-a2f2-4b36-b644-64d7ed7444b0-kube-api-access-sccp7\") pod \"38604d86-a2f2-4b36-b644-64d7ed7444b0\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") "
Nov 03 02:18:36.423182 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.422943    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-credential-socket\") pod \"38604d86-a2f2-4b36-b644-64d7ed7444b0\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") "
Nov 03 02:18:36.423182 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.422953    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-data" pod "38604d86-a2f2-4b36-b644-64d7ed7444b0" (UID: "38604d86-a2f2-4b36-b644-64d7ed7444b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:18:36.423182 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.423010    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-podinfo\") pod \"38604d86-a2f2-4b36-b644-64d7ed7444b0\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") "
Nov 03 02:18:36.423182 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.423139    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38604d86-a2f2-4b36-b644-64d7ed7444b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38604d86-a2f2-4b36-b644-64d7ed7444b0-istiod-ca-cert\") pod \"38604d86-a2f2-4b36-b644-64d7ed7444b0\" (UID: \"38604d86-a2f2-4b36-b644-64d7ed7444b0\") "
Nov 03 02:18:36.423805 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.423781    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:18:36.423982 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.423822    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-workload-certs" pod "38604d86-a2f2-4b36-b644-64d7ed7444b0" (UID: "38604d86-a2f2-4b36-b644-64d7ed7444b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:18:36.423982 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.423898    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-workload-socket" pod "38604d86-a2f2-4b36-b644-64d7ed7444b0" (UID: "38604d86-a2f2-4b36-b644-64d7ed7444b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:18:36.424406 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.424378    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-credential-socket" pod "38604d86-a2f2-4b36-b644-64d7ed7444b0" (UID: "38604d86-a2f2-4b36-b644-64d7ed7444b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:18:36.424651 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.424627    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38604d86-a2f2-4b36-b644-64d7ed7444b0-istiod-ca-cert" pod "38604d86-a2f2-4b36-b644-64d7ed7444b0" (UID: "38604d86-a2f2-4b36-b644-64d7ed7444b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:18:36.425982 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.425951    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-envoy" pod "38604d86-a2f2-4b36-b644-64d7ed7444b0" (UID: "38604d86-a2f2-4b36-b644-64d7ed7444b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:18:36.427264 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.427224    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38604d86-a2f2-4b36-b644-64d7ed7444b0-kube-api-access-sccp7" pod "38604d86-a2f2-4b36-b644-64d7ed7444b0" (UID: "38604d86-a2f2-4b36-b644-64d7ed7444b0"). InnerVolumeSpecName "kube-api-access-sccp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:18:36.427591 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.427569    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-token" pod "38604d86-a2f2-4b36-b644-64d7ed7444b0" (UID: "38604d86-a2f2-4b36-b644-64d7ed7444b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:18:36.427778 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.427756    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-podinfo" pod "38604d86-a2f2-4b36-b644-64d7ed7444b0" (UID: "38604d86-a2f2-4b36-b644-64d7ed7444b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:18:36.525108 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.525061    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:18:36.525108 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.525099    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38604d86-a2f2-4b36-b644-64d7ed7444b0-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:18:36.525108 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.525112    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:18:36.525371 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.525123    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:18:36.525371 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.525137    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:18:36.525371 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.525149    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:18:36.525371 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.525161    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sccp7\" (UniqueName: \"kubernetes.io/projected/38604d86-a2f2-4b36-b644-64d7ed7444b0-kube-api-access-sccp7\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:18:36.525371 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.525172    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38604d86-a2f2-4b36-b644-64d7ed7444b0-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:18:36.789941 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.789907    5005 generic.go:356] "Generic (PLEG): container finished" podID="38604d86-a2f2-4b36-b644-64d7ed7444b0" containerID="0b016549b9272958d4581ce4521cdd11c36de1253d306e13eab9a3a4a6e40681" exitCode=0
Nov 03 02:18:36.790150 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.789959    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"
Nov 03 02:18:36.790150 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.789984    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh" event={"ID":"38604d86-a2f2-4b36-b644-64d7ed7444b0","Type":"ContainerDied","Data":"0b016549b9272958d4581ce4521cdd11c36de1253d306e13eab9a3a4a6e40681"}
Nov 03 02:18:36.790150 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.790042    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh" event={"ID":"38604d86-a2f2-4b36-b644-64d7ed7444b0","Type":"ContainerDied","Data":"d470995b427bce80e757ee8f7a12c4c9debfb38d21c58f3106fab5f56058d389"}
Nov 03 02:18:36.790150 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.790069    5005 scope.go:122] "RemoveContainer" containerID="0b016549b9272958d4581ce4521cdd11c36de1253d306e13eab9a3a4a6e40681"
Nov 03 02:18:36.801420 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.801400    5005 scope.go:122] "RemoveContainer" containerID="0b016549b9272958d4581ce4521cdd11c36de1253d306e13eab9a3a4a6e40681"
Nov 03 02:18:36.801732 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:18:36.801709    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b016549b9272958d4581ce4521cdd11c36de1253d306e13eab9a3a4a6e40681\": container with ID starting with 0b016549b9272958d4581ce4521cdd11c36de1253d306e13eab9a3a4a6e40681 not found: ID does not exist" containerID="0b016549b9272958d4581ce4521cdd11c36de1253d306e13eab9a3a4a6e40681"
Nov 03 02:18:36.801810 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.801746    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b016549b9272958d4581ce4521cdd11c36de1253d306e13eab9a3a4a6e40681"} err="failed to get container status \"0b016549b9272958d4581ce4521cdd11c36de1253d306e13eab9a3a4a6e40681\": rpc error: code = NotFound desc = could not find container \"0b016549b9272958d4581ce4521cdd11c36de1253d306e13eab9a3a4a6e40681\": container with ID starting with 0b016549b9272958d4581ce4521cdd11c36de1253d306e13eab9a3a4a6e40681 not found: ID does not exist"
Nov 03 02:18:36.949562 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.946818    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"]
Nov 03 02:18:36.992023 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:36.991989    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-g5znh"]
Nov 03 02:18:37.110965 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:18:37.110871    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38604d86-a2f2-4b36-b644-64d7ed7444b0" path="/var/lib/kubelet/pods/38604d86-a2f2-4b36-b644-64d7ed7444b0/volumes"
Nov 03 02:19:00.715317 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:19:00.715231    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:19:28.748897 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.748862    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"]
Nov 03 02:19:28.749482 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749468    5005 cpu_manager.go:414] "containerMap: removing container" podUID="50e0635a-54b1-497e-9e2c-66f6961eee54" containerName="maas-api"
Nov 03 02:19:28.749525 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749483    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50e0635a-54b1-497e-9e2c-66f6961eee54" containerName="maas-api"
Nov 03 02:19:28.749525 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749497    5005 cpu_manager.go:414] "containerMap: removing container" podUID="1f8fac4d-a231-421f-993e-981c21ca4e81" containerName="maas-api"
Nov 03 02:19:28.749525 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749502    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f8fac4d-a231-421f-993e-981c21ca4e81" containerName="maas-api"
Nov 03 02:19:28.749525 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749513    5005 cpu_manager.go:414] "containerMap: removing container" podUID="2ac8ea76-22ba-4392-a97c-ca67dc13f427" containerName="istio-proxy"
Nov 03 02:19:28.749525 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749519    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ac8ea76-22ba-4392-a97c-ca67dc13f427" containerName="istio-proxy"
Nov 03 02:19:28.749783 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749529    5005 cpu_manager.go:414] "containerMap: removing container" podUID="b02bc9ee-d26b-48b2-b2ce-06784e778bce" containerName="maas-api"
Nov 03 02:19:28.749783 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749534    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02bc9ee-d26b-48b2-b2ce-06784e778bce" containerName="maas-api"
Nov 03 02:19:28.749783 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749552    5005 cpu_manager.go:414] "containerMap: removing container" podUID="c4637951-2a4e-4796-8b07-179acbdc0cfb" containerName="maas-api"
Nov 03 02:19:28.749783 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749556    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4637951-2a4e-4796-8b07-179acbdc0cfb" containerName="maas-api"
Nov 03 02:19:28.749783 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749567    5005 cpu_manager.go:414] "containerMap: removing container" podUID="55279a3b-4345-47ba-b065-b9ac01da611d" containerName="istio-proxy"
Nov 03 02:19:28.749783 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749571    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55279a3b-4345-47ba-b065-b9ac01da611d" containerName="istio-proxy"
Nov 03 02:19:28.749783 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749588    5005 cpu_manager.go:414] "containerMap: removing container" podUID="3ec530be-e064-476f-b3e3-6cafd2d3d1d4" containerName="maas-api"
Nov 03 02:19:28.749783 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749593    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ec530be-e064-476f-b3e3-6cafd2d3d1d4" containerName="maas-api"
Nov 03 02:19:28.749783 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749600    5005 cpu_manager.go:414] "containerMap: removing container" podUID="6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" containerName="istio-proxy"
Nov 03 02:19:28.749783 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749605    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" containerName="istio-proxy"
Nov 03 02:19:28.749783 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749614    5005 cpu_manager.go:414] "containerMap: removing container" podUID="38604d86-a2f2-4b36-b644-64d7ed7444b0" containerName="istio-proxy"
Nov 03 02:19:28.749783 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749618    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38604d86-a2f2-4b36-b644-64d7ed7444b0" containerName="istio-proxy"
Nov 03 02:19:28.749783 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749625    5005 cpu_manager.go:414] "containerMap: removing container" podUID="28e48d72-ce80-4a8e-ada5-c5f7f3cda02f" containerName="istio-proxy"
Nov 03 02:19:28.749783 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749629    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e48d72-ce80-4a8e-ada5-c5f7f3cda02f" containerName="istio-proxy"
Nov 03 02:19:28.749783 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749783    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ac8ea76-22ba-4392-a97c-ca67dc13f427" containerName="istio-proxy"
Nov 03 02:19:28.750347 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749798    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02bc9ee-d26b-48b2-b2ce-06784e778bce" containerName="maas-api"
Nov 03 02:19:28.750347 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749815    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4637951-2a4e-4796-8b07-179acbdc0cfb" containerName="maas-api"
Nov 03 02:19:28.750347 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749823    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="55279a3b-4345-47ba-b065-b9ac01da611d" containerName="istio-proxy"
Nov 03 02:19:28.750347 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749833    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ec530be-e064-476f-b3e3-6cafd2d3d1d4" containerName="maas-api"
Nov 03 02:19:28.750347 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749839    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f80420e-9ed1-46ea-9fb1-c5489c72ed4c" containerName="istio-proxy"
Nov 03 02:19:28.750347 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749846    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="38604d86-a2f2-4b36-b644-64d7ed7444b0" containerName="istio-proxy"
Nov 03 02:19:28.750347 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749852    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e48d72-ce80-4a8e-ada5-c5f7f3cda02f" containerName="istio-proxy"
Nov 03 02:19:28.750347 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749862    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="50e0635a-54b1-497e-9e2c-66f6961eee54" containerName="maas-api"
Nov 03 02:19:28.750347 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.749868    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f8fac4d-a231-421f-993e-981c21ca4e81" containerName="maas-api"
Nov 03 02:19:28.754638 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.754618    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"
Nov 03 02:19:28.807195 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.807162    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"]
Nov 03 02:19:28.849663 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.849620    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f66575-serving-cert\""
Nov 03 02:19:28.854507 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.854473    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8\" (UID: \"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"
Nov 03 02:19:28.854667 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.854615    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9gzt\" (UniqueName: \"kubernetes.io/projected/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-kube-api-access-t9gzt\") pod \"maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8\" (UID: \"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"
Nov 03 02:19:28.854748 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.854679    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8\" (UID: \"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"
Nov 03 02:19:28.854810 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.854794    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-metrics-certs\") pod \"maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8\" (UID: \"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"
Nov 03 02:19:28.955926 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.955885    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8\" (UID: \"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"
Nov 03 02:19:28.956128 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.955959    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9gzt\" (UniqueName: \"kubernetes.io/projected/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-kube-api-access-t9gzt\") pod \"maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8\" (UID: \"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"
Nov 03 02:19:28.956128 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.955983    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8\" (UID: \"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"
Nov 03 02:19:28.956128 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.956046    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-metrics-certs\") pod \"maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8\" (UID: \"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"
Nov 03 02:19:28.956306 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:19:28.956183    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-serving-cert: secret "maas-api-e2e-ipp-a-f66575-serving-cert" not found
Nov 03 02:19:28.956306 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:19:28.956268    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-maas-api-tls podName:1ee5f1b4-095d-42e3-84be-3a8b2e4400ee nodeName:}" failed. No retries permitted until 2026-11-03 02:19:29.456247085 +0000 UTC m=+1696.470488433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-maas-api-tls") pod "maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8" (UID: "1ee5f1b4-095d-42e3-84be-3a8b2e4400ee") : secret "maas-api-e2e-ipp-a-f66575-serving-cert" not found
Nov 03 02:19:28.956688 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.956664    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8\" (UID: \"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"
Nov 03 02:19:28.958506 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:28.958489    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-metrics-certs\") pod \"maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8\" (UID: \"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"
Nov 03 02:19:29.019063 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:29.019010    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9gzt\" (UniqueName: \"kubernetes.io/projected/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-kube-api-access-t9gzt\") pod \"maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8\" (UID: \"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"
Nov 03 02:19:29.467808 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:29.467689    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8\" (UID: \"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"
Nov 03 02:19:29.470194 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:29.470174    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8\" (UID: \"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"
Nov 03 02:19:29.668067 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:29.668026    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"
Nov 03 02:19:30.076077 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:30.075881    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"]
Nov 03 02:19:30.079625 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:19:30.079594    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee5f1b4_095d_42e3_84be_3a8b2e4400ee.slice/crio-3e4714fe774c453a01cc1eb88437ee39600297784c47467015c26a080f788c8b WatchSource:0}: Error finding container 3e4714fe774c453a01cc1eb88437ee39600297784c47467015c26a080f788c8b: Status 404 returned error can't find the container with id 3e4714fe774c453a01cc1eb88437ee39600297784c47467015c26a080f788c8b
Nov 03 02:19:30.081112 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:30.081091    5005 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 02:19:30.199861 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:30.199823    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8" event={"ID":"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee","Type":"ContainerStarted","Data":"3e4714fe774c453a01cc1eb88437ee39600297784c47467015c26a080f788c8b"}
Nov 03 02:19:30.713778 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:19:30.713741    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:19:32.212810 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:32.212779    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8" event={"ID":"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee","Type":"ContainerStarted","Data":"0cad936a8ba0266e0c7a002ca87734ccb7d0dd55b3fbddafcba63f13366bcb6f"}
Nov 03 02:19:32.213199 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:32.212894    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"
Nov 03 02:19:32.319851 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:32.319788    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8" podStartSLOduration=3.239380552 podStartE2EDuration="4.319770636s" totalImagesPullingTime="1.080390084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:19:32.319164793 +0000 UTC m=+1699.333406172" watchObservedRunningTime="2026-11-03 02:19:32.319770636 +0000 UTC m=+1699.334012006"
Nov 03 02:19:38.227070 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:38.227040    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"
Nov 03 02:19:50.008281 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.008245    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"]
Nov 03 02:19:50.008883 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.008474    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn" podUID="c0a3a84a-fd80-408e-854c-46f6f268c6fa" containerName="maas-api" containerID="cri-o://88ea407d596d64c8f336e8ba3e244b49a52c2e065d91377d3f34b90b5c21304f" gracePeriod=30
Nov 03 02:19:50.323217 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.323186    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"
Nov 03 02:19:50.374865 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.374832    5005 generic.go:356] "Generic (PLEG): container finished" podID="c0a3a84a-fd80-408e-854c-46f6f268c6fa" containerID="88ea407d596d64c8f336e8ba3e244b49a52c2e065d91377d3f34b90b5c21304f" exitCode=0
Nov 03 02:19:50.375041 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.374874    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"
Nov 03 02:19:50.375041 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.374915    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn" event={"ID":"c0a3a84a-fd80-408e-854c-46f6f268c6fa","Type":"ContainerDied","Data":"88ea407d596d64c8f336e8ba3e244b49a52c2e065d91377d3f34b90b5c21304f"}
Nov 03 02:19:50.375041 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.374968    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn" event={"ID":"c0a3a84a-fd80-408e-854c-46f6f268c6fa","Type":"ContainerDied","Data":"7639536a0d668cafd1f3c3ec45bd68eb602412601b87d24e0c0d82ad8d8ebd4d"}
Nov 03 02:19:50.375041 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.374991    5005 scope.go:122] "RemoveContainer" containerID="88ea407d596d64c8f336e8ba3e244b49a52c2e065d91377d3f34b90b5c21304f"
Nov 03 02:19:50.384666 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.384648    5005 scope.go:122] "RemoveContainer" containerID="88ea407d596d64c8f336e8ba3e244b49a52c2e065d91377d3f34b90b5c21304f"
Nov 03 02:19:50.384993 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:19:50.384966    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ea407d596d64c8f336e8ba3e244b49a52c2e065d91377d3f34b90b5c21304f\": container with ID starting with 88ea407d596d64c8f336e8ba3e244b49a52c2e065d91377d3f34b90b5c21304f not found: ID does not exist" containerID="88ea407d596d64c8f336e8ba3e244b49a52c2e065d91377d3f34b90b5c21304f"
Nov 03 02:19:50.385067 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.385006    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ea407d596d64c8f336e8ba3e244b49a52c2e065d91377d3f34b90b5c21304f"} err="failed to get container status \"88ea407d596d64c8f336e8ba3e244b49a52c2e065d91377d3f34b90b5c21304f\": rpc error: code = NotFound desc = could not find container \"88ea407d596d64c8f336e8ba3e244b49a52c2e065d91377d3f34b90b5c21304f\": container with ID starting with 88ea407d596d64c8f336e8ba3e244b49a52c2e065d91377d3f34b90b5c21304f not found: ID does not exist"
Nov 03 02:19:50.412888 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.412843    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a3a84a-fd80-408e-854c-46f6f268c6fa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0a3a84a-fd80-408e-854c-46f6f268c6fa-maas-api-tls\") pod \"c0a3a84a-fd80-408e-854c-46f6f268c6fa\" (UID: \"c0a3a84a-fd80-408e-854c-46f6f268c6fa\") "
Nov 03 02:19:50.413250 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.413225    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a3a84a-fd80-408e-854c-46f6f268c6fa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a3a84a-fd80-408e-854c-46f6f268c6fa-metrics-certs\") pod \"c0a3a84a-fd80-408e-854c-46f6f268c6fa\" (UID: \"c0a3a84a-fd80-408e-854c-46f6f268c6fa\") "
Nov 03 02:19:50.413666 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.413643    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a3a84a-fd80-408e-854c-46f6f268c6fa-kube-api-access-ch6tl\" (UniqueName: \"kubernetes.io/projected/c0a3a84a-fd80-408e-854c-46f6f268c6fa-kube-api-access-ch6tl\") pod \"c0a3a84a-fd80-408e-854c-46f6f268c6fa\" (UID: \"c0a3a84a-fd80-408e-854c-46f6f268c6fa\") "
Nov 03 02:19:50.413940 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.413915    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0a3a84a-fd80-408e-854c-46f6f268c6fa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a3a84a-fd80-408e-854c-46f6f268c6fa-openshift-service-ca\") pod \"c0a3a84a-fd80-408e-854c-46f6f268c6fa\" (UID: \"c0a3a84a-fd80-408e-854c-46f6f268c6fa\") "
Nov 03 02:19:50.414837 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.414797    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a3a84a-fd80-408e-854c-46f6f268c6fa-openshift-service-ca" pod "c0a3a84a-fd80-408e-854c-46f6f268c6fa" (UID: "c0a3a84a-fd80-408e-854c-46f6f268c6fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:19:50.428179 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.428018    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a3a84a-fd80-408e-854c-46f6f268c6fa-kube-api-access-ch6tl" pod "c0a3a84a-fd80-408e-854c-46f6f268c6fa" (UID: "c0a3a84a-fd80-408e-854c-46f6f268c6fa"). InnerVolumeSpecName "kube-api-access-ch6tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:19:50.428179 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.428104    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a3a84a-fd80-408e-854c-46f6f268c6fa-maas-api-tls" pod "c0a3a84a-fd80-408e-854c-46f6f268c6fa" (UID: "c0a3a84a-fd80-408e-854c-46f6f268c6fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:19:50.468749 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.460831    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a3a84a-fd80-408e-854c-46f6f268c6fa-metrics-certs" pod "c0a3a84a-fd80-408e-854c-46f6f268c6fa" (UID: "c0a3a84a-fd80-408e-854c-46f6f268c6fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:19:50.516436 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.516403    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0a3a84a-fd80-408e-854c-46f6f268c6fa-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:19:50.516436 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.516434    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a3a84a-fd80-408e-854c-46f6f268c6fa-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:19:50.516436 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.516444    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch6tl\" (UniqueName: \"kubernetes.io/projected/c0a3a84a-fd80-408e-854c-46f6f268c6fa-kube-api-access-ch6tl\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:19:50.516678 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.516454    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a3a84a-fd80-408e-854c-46f6f268c6fa-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:19:50.791859 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.791799    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"]
Nov 03 02:19:50.798653 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:50.798621    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6198c6-75774b485c-swdtn"]
Nov 03 02:19:51.153984 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:19:51.153945    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0a3a84a-fd80-408e-854c-46f6f268c6fa" path="/var/lib/kubelet/pods/c0a3a84a-fd80-408e-854c-46f6f268c6fa/volumes"
Nov 03 02:20:00.718424 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:20:00.718380    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:20:30.714245 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:20:30.714206    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:20:50.342807 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:50.342768    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"]
Nov 03 02:20:50.343340 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:50.342997    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w" podUID="06335e60-e878-4ad3-9630-fabee17ef5d9" containerName="istio-proxy" containerID="cri-o://30b11792a4a028a500cfd2133f9ac904fce4278eca4ceff2993cb43f497b7212" gracePeriod=30
Nov 03 02:20:52.494860 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:52.494820    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 02:20:52.495334 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:52.494896    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w" podUID="06335e60-e878-4ad3-9630-fabee17ef5d9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 02:20:55.632884 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.632814    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:20:55.729988 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.727761    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-token\" (UniqueName: \"kubernetes.io/projected/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-token\") pod \"06335e60-e878-4ad3-9630-fabee17ef5d9\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") "
Nov 03 02:20:55.729988 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.727814    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-workload-certs\") pod \"06335e60-e878-4ad3-9630-fabee17ef5d9\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") "
Nov 03 02:20:55.729988 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.727863    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-podinfo\") pod \"06335e60-e878-4ad3-9630-fabee17ef5d9\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") "
Nov 03 02:20:55.729988 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.728013    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-envoy\") pod \"06335e60-e878-4ad3-9630-fabee17ef5d9\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") "
Nov 03 02:20:55.729988 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.728056    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-credential-socket\") pod \"06335e60-e878-4ad3-9630-fabee17ef5d9\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") "
Nov 03 02:20:55.729988 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.728170    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06335e60-e878-4ad3-9630-fabee17ef5d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06335e60-e878-4ad3-9630-fabee17ef5d9-istiod-ca-cert\") pod \"06335e60-e878-4ad3-9630-fabee17ef5d9\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") "
Nov 03 02:20:55.729988 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.728200    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-data\") pod \"06335e60-e878-4ad3-9630-fabee17ef5d9\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") "
Nov 03 02:20:55.729988 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.728280    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-workload-socket\") pod \"06335e60-e878-4ad3-9630-fabee17ef5d9\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") "
Nov 03 02:20:55.729988 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.728361    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06335e60-e878-4ad3-9630-fabee17ef5d9-kube-api-access-bzzqg\" (UniqueName: \"kubernetes.io/projected/06335e60-e878-4ad3-9630-fabee17ef5d9-kube-api-access-bzzqg\") pod \"06335e60-e878-4ad3-9630-fabee17ef5d9\" (UID: \"06335e60-e878-4ad3-9630-fabee17ef5d9\") "
Nov 03 02:20:55.729988 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.729203    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-workload-certs" pod "06335e60-e878-4ad3-9630-fabee17ef5d9" (UID: "06335e60-e878-4ad3-9630-fabee17ef5d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:20:55.729988 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.729465    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-credential-socket" pod "06335e60-e878-4ad3-9630-fabee17ef5d9" (UID: "06335e60-e878-4ad3-9630-fabee17ef5d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:20:55.729988 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.729664    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-workload-socket" pod "06335e60-e878-4ad3-9630-fabee17ef5d9" (UID: "06335e60-e878-4ad3-9630-fabee17ef5d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:20:55.729988 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.729906    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06335e60-e878-4ad3-9630-fabee17ef5d9-istiod-ca-cert" pod "06335e60-e878-4ad3-9630-fabee17ef5d9" (UID: "06335e60-e878-4ad3-9630-fabee17ef5d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:20:55.729988 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.729610    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-data" pod "06335e60-e878-4ad3-9630-fabee17ef5d9" (UID: "06335e60-e878-4ad3-9630-fabee17ef5d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:20:55.731488 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.731461    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-envoy" pod "06335e60-e878-4ad3-9630-fabee17ef5d9" (UID: "06335e60-e878-4ad3-9630-fabee17ef5d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:20:55.732082 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.732052    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-token" pod "06335e60-e878-4ad3-9630-fabee17ef5d9" (UID: "06335e60-e878-4ad3-9630-fabee17ef5d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:20:55.733070 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.733046    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-podinfo" pod "06335e60-e878-4ad3-9630-fabee17ef5d9" (UID: "06335e60-e878-4ad3-9630-fabee17ef5d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:20:55.733203 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.733184    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06335e60-e878-4ad3-9630-fabee17ef5d9-kube-api-access-bzzqg" pod "06335e60-e878-4ad3-9630-fabee17ef5d9" (UID: "06335e60-e878-4ad3-9630-fabee17ef5d9"). InnerVolumeSpecName "kube-api-access-bzzqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:20:55.833138 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.832972    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:20:55.833138 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.833010    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:20:55.833138 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.833026    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06335e60-e878-4ad3-9630-fabee17ef5d9-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:20:55.833138 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.833039    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:20:55.833138 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.833053    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:20:55.833138 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.833065    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzzqg\" (UniqueName: \"kubernetes.io/projected/06335e60-e878-4ad3-9630-fabee17ef5d9-kube-api-access-bzzqg\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:20:55.833138 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.833078    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:20:55.833138 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.833090    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06335e60-e878-4ad3-9630-fabee17ef5d9-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:20:55.833138 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.833102    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06335e60-e878-4ad3-9630-fabee17ef5d9-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:20:55.865940 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.865900    5005 generic.go:356] "Generic (PLEG): container finished" podID="06335e60-e878-4ad3-9630-fabee17ef5d9" containerID="30b11792a4a028a500cfd2133f9ac904fce4278eca4ceff2993cb43f497b7212" exitCode=0
Nov 03 02:20:55.866155 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.865953    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"
Nov 03 02:20:55.866155 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.866008    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w" event={"ID":"06335e60-e878-4ad3-9630-fabee17ef5d9","Type":"ContainerDied","Data":"30b11792a4a028a500cfd2133f9ac904fce4278eca4ceff2993cb43f497b7212"}
Nov 03 02:20:55.866489 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.866467    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w" event={"ID":"06335e60-e878-4ad3-9630-fabee17ef5d9","Type":"ContainerDied","Data":"e121ceecff4c74daf716e7882d6f018d5d0698b4422feb534c9da94132e1ae0e"}
Nov 03 02:20:55.866627 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.866512    5005 scope.go:122] "RemoveContainer" containerID="30b11792a4a028a500cfd2133f9ac904fce4278eca4ceff2993cb43f497b7212"
Nov 03 02:20:55.899948 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.898080    5005 scope.go:122] "RemoveContainer" containerID="30b11792a4a028a500cfd2133f9ac904fce4278eca4ceff2993cb43f497b7212"
Nov 03 02:20:55.899948 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:20:55.899555    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30b11792a4a028a500cfd2133f9ac904fce4278eca4ceff2993cb43f497b7212\": container with ID starting with 30b11792a4a028a500cfd2133f9ac904fce4278eca4ceff2993cb43f497b7212 not found: ID does not exist" containerID="30b11792a4a028a500cfd2133f9ac904fce4278eca4ceff2993cb43f497b7212"
Nov 03 02:20:55.899948 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:55.899600    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30b11792a4a028a500cfd2133f9ac904fce4278eca4ceff2993cb43f497b7212"} err="failed to get container status \"30b11792a4a028a500cfd2133f9ac904fce4278eca4ceff2993cb43f497b7212\": rpc error: code = NotFound desc = could not find container \"30b11792a4a028a500cfd2133f9ac904fce4278eca4ceff2993cb43f497b7212\": container with ID starting with 30b11792a4a028a500cfd2133f9ac904fce4278eca4ceff2993cb43f497b7212 not found: ID does not exist"
Nov 03 02:20:56.023521 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:56.023484    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"]
Nov 03 02:20:56.027239 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:56.027213    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-9477w"]
Nov 03 02:20:57.117587 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:20:57.117544    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06335e60-e878-4ad3-9630-fabee17ef5d9" path="/var/lib/kubelet/pods/06335e60-e878-4ad3-9630-fabee17ef5d9/volumes"
Nov 03 02:21:00.154309 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.154266    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"]
Nov 03 02:21:00.154895 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.154876    5005 cpu_manager.go:414] "containerMap: removing container" podUID="06335e60-e878-4ad3-9630-fabee17ef5d9" containerName="istio-proxy"
Nov 03 02:21:00.154961 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.154897    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06335e60-e878-4ad3-9630-fabee17ef5d9" containerName="istio-proxy"
Nov 03 02:21:00.154961 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.154948    5005 cpu_manager.go:414] "containerMap: removing container" podUID="c0a3a84a-fd80-408e-854c-46f6f268c6fa" containerName="maas-api"
Nov 03 02:21:00.154961 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.154954    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a3a84a-fd80-408e-854c-46f6f268c6fa" containerName="maas-api"
Nov 03 02:21:00.155076 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.155060    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="06335e60-e878-4ad3-9630-fabee17ef5d9" containerName="istio-proxy"
Nov 03 02:21:00.155076 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.155075    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a3a84a-fd80-408e-854c-46f6f268c6fa" containerName="maas-api"
Nov 03 02:21:00.159991 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.159972    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"
Nov 03 02:21:00.213119 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.213088    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"]
Nov 03 02:21:00.258609 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.258568    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3bbfc4-serving-cert\""
Nov 03 02:21:00.299645 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.299600    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88c68e02-3246-440d-b299-ab6c10bf4258-maas-api-tls\") pod \"maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d\" (UID: \"88c68e02-3246-440d-b299-ab6c10bf4258\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"
Nov 03 02:21:00.299856 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.299685    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88c68e02-3246-440d-b299-ab6c10bf4258-openshift-service-ca\") pod \"maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d\" (UID: \"88c68e02-3246-440d-b299-ab6c10bf4258\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"
Nov 03 02:21:00.299856 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.299846    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4ckb\" (UniqueName: \"kubernetes.io/projected/88c68e02-3246-440d-b299-ab6c10bf4258-kube-api-access-j4ckb\") pod \"maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d\" (UID: \"88c68e02-3246-440d-b299-ab6c10bf4258\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"
Nov 03 02:21:00.300030 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.299993    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c68e02-3246-440d-b299-ab6c10bf4258-metrics-certs\") pod \"maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d\" (UID: \"88c68e02-3246-440d-b299-ab6c10bf4258\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"
Nov 03 02:21:00.403729 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.403604    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4ckb\" (UniqueName: \"kubernetes.io/projected/88c68e02-3246-440d-b299-ab6c10bf4258-kube-api-access-j4ckb\") pod \"maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d\" (UID: \"88c68e02-3246-440d-b299-ab6c10bf4258\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"
Nov 03 02:21:00.404018 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.403836    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c68e02-3246-440d-b299-ab6c10bf4258-metrics-certs\") pod \"maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d\" (UID: \"88c68e02-3246-440d-b299-ab6c10bf4258\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"
Nov 03 02:21:00.404018 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.403912    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88c68e02-3246-440d-b299-ab6c10bf4258-maas-api-tls\") pod \"maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d\" (UID: \"88c68e02-3246-440d-b299-ab6c10bf4258\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"
Nov 03 02:21:00.404018 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.403966    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88c68e02-3246-440d-b299-ab6c10bf4258-openshift-service-ca\") pod \"maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d\" (UID: \"88c68e02-3246-440d-b299-ab6c10bf4258\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"
Nov 03 02:21:00.410159 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.410082    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88c68e02-3246-440d-b299-ab6c10bf4258-openshift-service-ca\") pod \"maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d\" (UID: \"88c68e02-3246-440d-b299-ab6c10bf4258\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"
Nov 03 02:21:00.410986 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:21:00.410962    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-serving-cert: secret "maas-api-e2e-api-a-3bbfc4-serving-cert" not found
Nov 03 02:21:00.411250 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:21:00.411230    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88c68e02-3246-440d-b299-ab6c10bf4258-maas-api-tls podName:88c68e02-3246-440d-b299-ab6c10bf4258 nodeName:}" failed. No retries permitted until 2026-11-03 02:21:00.911207651 +0000 UTC m=+1787.925448994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/88c68e02-3246-440d-b299-ab6c10bf4258-maas-api-tls") pod "maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d" (UID: "88c68e02-3246-440d-b299-ab6c10bf4258") : secret "maas-api-e2e-api-a-3bbfc4-serving-cert" not found
Nov 03 02:21:00.416780 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.416749    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c68e02-3246-440d-b299-ab6c10bf4258-metrics-certs\") pod \"maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d\" (UID: \"88c68e02-3246-440d-b299-ab6c10bf4258\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"
Nov 03 02:21:00.465478 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.465443    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4ckb\" (UniqueName: \"kubernetes.io/projected/88c68e02-3246-440d-b299-ab6c10bf4258-kube-api-access-j4ckb\") pod \"maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d\" (UID: \"88c68e02-3246-440d-b299-ab6c10bf4258\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"
Nov 03 02:21:00.716379 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:21:00.716289    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:21:00.911781 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.911735    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88c68e02-3246-440d-b299-ab6c10bf4258-maas-api-tls\") pod \"maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d\" (UID: \"88c68e02-3246-440d-b299-ab6c10bf4258\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"
Nov 03 02:21:00.914277 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:00.914243    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88c68e02-3246-440d-b299-ab6c10bf4258-maas-api-tls\") pod \"maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d\" (UID: \"88c68e02-3246-440d-b299-ab6c10bf4258\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"
Nov 03 02:21:01.073603 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:01.073564    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"
Nov 03 02:21:01.452085 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:01.452000    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"]
Nov 03 02:21:01.457997 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:21:01.457967    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c68e02_3246_440d_b299_ab6c10bf4258.slice/crio-e42d7fa4c19b33d244b74190f82719850eb009ccd3a467bb729708550e3ed410 WatchSource:0}: Error finding container e42d7fa4c19b33d244b74190f82719850eb009ccd3a467bb729708550e3ed410: Status 404 returned error can't find the container with id e42d7fa4c19b33d244b74190f82719850eb009ccd3a467bb729708550e3ed410
Nov 03 02:21:01.906763 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:01.906723    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d" event={"ID":"88c68e02-3246-440d-b299-ab6c10bf4258","Type":"ContainerStarted","Data":"e42d7fa4c19b33d244b74190f82719850eb009ccd3a467bb729708550e3ed410"}
Nov 03 02:21:02.914564 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:02.914530    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d" event={"ID":"88c68e02-3246-440d-b299-ab6c10bf4258","Type":"ContainerStarted","Data":"120d801c49f54929c7928596296b7def6476bbf97cb98da6a218397d78344c58"}
Nov 03 02:21:02.915017 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:02.914613    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"
Nov 03 02:21:03.024871 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:03.024815    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d" podStartSLOduration=2.186037576 podStartE2EDuration="3.024799563s" totalImagesPullingTime="838.761987ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:21:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:21:03.0209184 +0000 UTC m=+1790.035159766" watchObservedRunningTime="2026-11-03 02:21:03.024799563 +0000 UTC m=+1790.039040928"
Nov 03 02:21:08.926543 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:08.926509    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"
Nov 03 02:21:13.353981 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:13.353950    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:21:13.366487 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:13.366460    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:21:30.715303 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:21:30.715258    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:21:41.637964 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:41.637927    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"]
Nov 03 02:21:41.642076 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:41.642055    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"
Nov 03 02:21:41.696045 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:41.696007    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"]
Nov 03 02:21:41.701364 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:41.701327    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30ce4650-88d3-4c9e-a333-7fdbea243a40-metrics-certs\") pod \"maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl\" (UID: \"30ce4650-88d3-4c9e-a333-7fdbea243a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"
Nov 03 02:21:41.701725 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:41.701668    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tddlf\" (UniqueName: \"kubernetes.io/projected/30ce4650-88d3-4c9e-a333-7fdbea243a40-kube-api-access-tddlf\") pod \"maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl\" (UID: \"30ce4650-88d3-4c9e-a333-7fdbea243a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"
Nov 03 02:21:41.701844 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:41.701735    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ce4650-88d3-4c9e-a333-7fdbea243a40-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl\" (UID: \"30ce4650-88d3-4c9e-a333-7fdbea243a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"
Nov 03 02:21:41.701844 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:41.701823    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ce4650-88d3-4c9e-a333-7fdbea243a40-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl\" (UID: \"30ce4650-88d3-4c9e-a333-7fdbea243a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"
Nov 03 02:21:41.738310 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:41.738274    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-9bd3ab-serving-cert\""
Nov 03 02:21:41.802744 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:41.802709    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30ce4650-88d3-4c9e-a333-7fdbea243a40-metrics-certs\") pod \"maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl\" (UID: \"30ce4650-88d3-4c9e-a333-7fdbea243a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"
Nov 03 02:21:41.802925 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:41.802803    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tddlf\" (UniqueName: \"kubernetes.io/projected/30ce4650-88d3-4c9e-a333-7fdbea243a40-kube-api-access-tddlf\") pod \"maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl\" (UID: \"30ce4650-88d3-4c9e-a333-7fdbea243a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"
Nov 03 02:21:41.802925 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:41.802846    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ce4650-88d3-4c9e-a333-7fdbea243a40-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl\" (UID: \"30ce4650-88d3-4c9e-a333-7fdbea243a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"
Nov 03 02:21:41.802925 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:41.802898    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ce4650-88d3-4c9e-a333-7fdbea243a40-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl\" (UID: \"30ce4650-88d3-4c9e-a333-7fdbea243a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"
Nov 03 02:21:41.803118 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:21:41.802983    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-serving-cert: secret "maas-api-e2e-ipp-b-9bd3ab-serving-cert" not found
Nov 03 02:21:41.803118 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:21:41.803074    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30ce4650-88d3-4c9e-a333-7fdbea243a40-maas-api-tls podName:30ce4650-88d3-4c9e-a333-7fdbea243a40 nodeName:}" failed. No retries permitted until 2026-11-03 02:21:42.303052232 +0000 UTC m=+1829.317293592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/30ce4650-88d3-4c9e-a333-7fdbea243a40-maas-api-tls") pod "maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl" (UID: "30ce4650-88d3-4c9e-a333-7fdbea243a40") : secret "maas-api-e2e-ipp-b-9bd3ab-serving-cert" not found
Nov 03 02:21:41.803815 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:41.803770    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ce4650-88d3-4c9e-a333-7fdbea243a40-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl\" (UID: \"30ce4650-88d3-4c9e-a333-7fdbea243a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"
Nov 03 02:21:41.805807 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:41.805786    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30ce4650-88d3-4c9e-a333-7fdbea243a40-metrics-certs\") pod \"maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl\" (UID: \"30ce4650-88d3-4c9e-a333-7fdbea243a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"
Nov 03 02:21:41.857461 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:41.857420    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tddlf\" (UniqueName: \"kubernetes.io/projected/30ce4650-88d3-4c9e-a333-7fdbea243a40-kube-api-access-tddlf\") pod \"maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl\" (UID: \"30ce4650-88d3-4c9e-a333-7fdbea243a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"
Nov 03 02:21:42.309422 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:42.309382    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ce4650-88d3-4c9e-a333-7fdbea243a40-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl\" (UID: \"30ce4650-88d3-4c9e-a333-7fdbea243a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"
Nov 03 02:21:42.312113 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:42.312084    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ce4650-88d3-4c9e-a333-7fdbea243a40-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl\" (UID: \"30ce4650-88d3-4c9e-a333-7fdbea243a40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"
Nov 03 02:21:42.556515 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:42.556479    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"
Nov 03 02:21:42.937816 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:42.937724    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"]
Nov 03 02:21:42.942992 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:21:42.942957    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ce4650_88d3_4c9e_a333_7fdbea243a40.slice/crio-b001ab33c6a86ef350226502eea67c7ff88ef880b27c625d68f7e06f0de2ca09 WatchSource:0}: Error finding container b001ab33c6a86ef350226502eea67c7ff88ef880b27c625d68f7e06f0de2ca09: Status 404 returned error can't find the container with id b001ab33c6a86ef350226502eea67c7ff88ef880b27c625d68f7e06f0de2ca09
Nov 03 02:21:43.228509 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:43.228419    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl" event={"ID":"30ce4650-88d3-4c9e-a333-7fdbea243a40","Type":"ContainerStarted","Data":"b001ab33c6a86ef350226502eea67c7ff88ef880b27c625d68f7e06f0de2ca09"}
Nov 03 02:21:44.235944 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:44.235851    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl" event={"ID":"30ce4650-88d3-4c9e-a333-7fdbea243a40","Type":"ContainerStarted","Data":"a8e801f5f0b62a02a7bdfb9cf3226c70f632ed7482b871029b37d51452022572"}
Nov 03 02:21:44.235944 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:44.235921    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"
Nov 03 02:21:44.343556 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:44.343502    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl" podStartSLOduration=2.480104587 podStartE2EDuration="3.343488188s" totalImagesPullingTime="863.383601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:21:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:21:44.341856232 +0000 UTC m=+1831.356097599" watchObservedRunningTime="2026-11-03 02:21:44.343488188 +0000 UTC m=+1831.357729555"
Nov 03 02:21:50.247400 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:50.247368    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"
Nov 03 02:21:55.955078 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:55.954993    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"]
Nov 03 02:21:55.959573 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:55.959550    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"
Nov 03 02:21:56.056524 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:56.056484    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-ad4591-serving-cert\""
Nov 03 02:21:56.062335 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:56.062303    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"]
Nov 03 02:21:56.091612 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:56.091579    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a205d365-889f-4706-9967-3e98bc1fc7d2-metrics-certs\") pod \"maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8\" (UID: \"a205d365-889f-4706-9967-3e98bc1fc7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"
Nov 03 02:21:56.091859 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:56.091730    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbwj9\" (UniqueName: \"kubernetes.io/projected/a205d365-889f-4706-9967-3e98bc1fc7d2-kube-api-access-dbwj9\") pod \"maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8\" (UID: \"a205d365-889f-4706-9967-3e98bc1fc7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"
Nov 03 02:21:56.091859 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:56.091840    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a205d365-889f-4706-9967-3e98bc1fc7d2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8\" (UID: \"a205d365-889f-4706-9967-3e98bc1fc7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"
Nov 03 02:21:56.092005 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:56.091875    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a205d365-889f-4706-9967-3e98bc1fc7d2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8\" (UID: \"a205d365-889f-4706-9967-3e98bc1fc7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"
Nov 03 02:21:56.193769 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:56.193735    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbwj9\" (UniqueName: \"kubernetes.io/projected/a205d365-889f-4706-9967-3e98bc1fc7d2-kube-api-access-dbwj9\") pod \"maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8\" (UID: \"a205d365-889f-4706-9967-3e98bc1fc7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"
Nov 03 02:21:56.193991 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:56.193829    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a205d365-889f-4706-9967-3e98bc1fc7d2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8\" (UID: \"a205d365-889f-4706-9967-3e98bc1fc7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"
Nov 03 02:21:56.193991 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:56.193856    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a205d365-889f-4706-9967-3e98bc1fc7d2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8\" (UID: \"a205d365-889f-4706-9967-3e98bc1fc7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"
Nov 03 02:21:56.193991 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:56.193949    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a205d365-889f-4706-9967-3e98bc1fc7d2-metrics-certs\") pod \"maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8\" (UID: \"a205d365-889f-4706-9967-3e98bc1fc7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"
Nov 03 02:21:56.193991 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:21:56.193967    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-serving-cert: secret "maas-api-e2e-ipp-b-ad4591-serving-cert" not found
Nov 03 02:21:56.194268 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:21:56.194059    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a205d365-889f-4706-9967-3e98bc1fc7d2-maas-api-tls podName:a205d365-889f-4706-9967-3e98bc1fc7d2 nodeName:}" failed. No retries permitted until 2026-11-03 02:21:56.69403487 +0000 UTC m=+1843.708276214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a205d365-889f-4706-9967-3e98bc1fc7d2-maas-api-tls") pod "maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8" (UID: "a205d365-889f-4706-9967-3e98bc1fc7d2") : secret "maas-api-e2e-ipp-b-ad4591-serving-cert" not found
Nov 03 02:21:56.194738 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:56.194712    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a205d365-889f-4706-9967-3e98bc1fc7d2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8\" (UID: \"a205d365-889f-4706-9967-3e98bc1fc7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"
Nov 03 02:21:56.197495 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:56.197466    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a205d365-889f-4706-9967-3e98bc1fc7d2-metrics-certs\") pod \"maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8\" (UID: \"a205d365-889f-4706-9967-3e98bc1fc7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"
Nov 03 02:21:56.248846 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:56.248742    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbwj9\" (UniqueName: \"kubernetes.io/projected/a205d365-889f-4706-9967-3e98bc1fc7d2-kube-api-access-dbwj9\") pod \"maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8\" (UID: \"a205d365-889f-4706-9967-3e98bc1fc7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"
Nov 03 02:21:56.701273 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:56.701229    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a205d365-889f-4706-9967-3e98bc1fc7d2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8\" (UID: \"a205d365-889f-4706-9967-3e98bc1fc7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"
Nov 03 02:21:56.703889 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:56.703862    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a205d365-889f-4706-9967-3e98bc1fc7d2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8\" (UID: \"a205d365-889f-4706-9967-3e98bc1fc7d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"
Nov 03 02:21:56.873011 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:56.872975    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"
Nov 03 02:21:57.247142 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:57.247101    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"]
Nov 03 02:21:57.252436 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:21:57.252403    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda205d365_889f_4706_9967_3e98bc1fc7d2.slice/crio-180391ed58ef5524e718a76364c8c86525abbf349741d5faf150111c70a2410b WatchSource:0}: Error finding container 180391ed58ef5524e718a76364c8c86525abbf349741d5faf150111c70a2410b: Status 404 returned error can't find the container with id 180391ed58ef5524e718a76364c8c86525abbf349741d5faf150111c70a2410b
Nov 03 02:21:57.319807 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:57.319768    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8" event={"ID":"a205d365-889f-4706-9967-3e98bc1fc7d2","Type":"ContainerStarted","Data":"180391ed58ef5524e718a76364c8c86525abbf349741d5faf150111c70a2410b"}
Nov 03 02:21:58.327836 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:58.327804    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8" event={"ID":"a205d365-889f-4706-9967-3e98bc1fc7d2","Type":"ContainerStarted","Data":"4dc46c05ba2c9aa3555a0554dafc4e7ce8d05a8346bd42cb042e72a7c27448a4"}
Nov 03 02:21:58.328240 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:58.327878    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"
Nov 03 02:21:58.437332 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:21:58.437229    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8" podStartSLOduration=2.553750779 podStartE2EDuration="3.437214503s" totalImagesPullingTime="883.463724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:21:58.433938114 +0000 UTC m=+1845.448179481" watchObservedRunningTime="2026-11-03 02:21:58.437214503 +0000 UTC m=+1845.451455878"
Nov 03 02:22:00.714481 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:22:00.714436    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:22:04.340725 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:04.340677    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"
Nov 03 02:22:22.374997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:22.374946    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"]
Nov 03 02:22:22.375562 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:22.375167    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl" podUID="00145046-700c-436b-a729-4332d8935371" containerName="istio-proxy" containerID="cri-o://24ec2c025816d2a18e1595dfd26e3cb3a74ab4e2ca6e8b525ba9131025414df0" gracePeriod=30
Nov 03 02:22:25.709148 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:25.709111    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 02:22:25.709537 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:25.709178    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl" podUID="00145046-700c-436b-a729-4332d8935371" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 02:22:25.831009 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:25.830971    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"]
Nov 03 02:22:25.835559 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:25.835538    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"
Nov 03 02:22:25.884458 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:25.884421    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-56299c-serving-cert\""
Nov 03 02:22:25.889086 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:25.889054    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"]
Nov 03 02:22:25.900139 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:25.900111    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-metrics-certs\") pod \"maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk\" (UID: \"7ebd4e3a-53a6-414a-94a2-726e9815b7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"
Nov 03 02:22:25.900311 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:25.900172    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5sw8\" (UniqueName: \"kubernetes.io/projected/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-kube-api-access-m5sw8\") pod \"maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk\" (UID: \"7ebd4e3a-53a6-414a-94a2-726e9815b7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"
Nov 03 02:22:25.900388 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:25.900329    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-maas-api-tls\") pod \"maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk\" (UID: \"7ebd4e3a-53a6-414a-94a2-726e9815b7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"
Nov 03 02:22:25.900388 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:25.900370    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk\" (UID: \"7ebd4e3a-53a6-414a-94a2-726e9815b7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"
Nov 03 02:22:26.001983 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:26.001954    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-metrics-certs\") pod \"maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk\" (UID: \"7ebd4e3a-53a6-414a-94a2-726e9815b7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"
Nov 03 02:22:26.002150 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:26.002034    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5sw8\" (UniqueName: \"kubernetes.io/projected/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-kube-api-access-m5sw8\") pod \"maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk\" (UID: \"7ebd4e3a-53a6-414a-94a2-726e9815b7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"
Nov 03 02:22:26.002150 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:26.002091    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-maas-api-tls\") pod \"maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk\" (UID: \"7ebd4e3a-53a6-414a-94a2-726e9815b7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"
Nov 03 02:22:26.002150 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:26.002114    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk\" (UID: \"7ebd4e3a-53a6-414a-94a2-726e9815b7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"
Nov 03 02:22:26.002323 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:22:26.002194    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-serving-cert: secret "maas-api-e2e-ipp-a-56299c-serving-cert" not found
Nov 03 02:22:26.002323 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:22:26.002277    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-maas-api-tls podName:7ebd4e3a-53a6-414a-94a2-726e9815b7aa nodeName:}" failed. No retries permitted until 2026-11-03 02:22:26.502254442 +0000 UTC m=+1873.516495789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-maas-api-tls") pod "maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk" (UID: "7ebd4e3a-53a6-414a-94a2-726e9815b7aa") : secret "maas-api-e2e-ipp-a-56299c-serving-cert" not found
Nov 03 02:22:26.002839 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:26.002815    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk\" (UID: \"7ebd4e3a-53a6-414a-94a2-726e9815b7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"
Nov 03 02:22:26.004485 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:26.004463    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-metrics-certs\") pod \"maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk\" (UID: \"7ebd4e3a-53a6-414a-94a2-726e9815b7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"
Nov 03 02:22:26.055796 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:26.055764    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5sw8\" (UniqueName: \"kubernetes.io/projected/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-kube-api-access-m5sw8\") pod \"maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk\" (UID: \"7ebd4e3a-53a6-414a-94a2-726e9815b7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"
Nov 03 02:22:26.508540 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:26.508499    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-maas-api-tls\") pod \"maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk\" (UID: \"7ebd4e3a-53a6-414a-94a2-726e9815b7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"
Nov 03 02:22:26.511019 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:26.510993    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-maas-api-tls\") pod \"maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk\" (UID: \"7ebd4e3a-53a6-414a-94a2-726e9815b7aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"
Nov 03 02:22:26.750672 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:26.750641    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"
Nov 03 02:22:27.131881 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.131804    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"]
Nov 03 02:22:27.137775 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:22:27.137744    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebd4e3a_53a6_414a_94a2_726e9815b7aa.slice/crio-c2d5af2cf99f0bf0a9c915364e5a4f22b78c013227d939823981b2350057dfb1 WatchSource:0}: Error finding container c2d5af2cf99f0bf0a9c915364e5a4f22b78c013227d939823981b2350057dfb1: Status 404 returned error can't find the container with id c2d5af2cf99f0bf0a9c915364e5a4f22b78c013227d939823981b2350057dfb1
Nov 03 02:22:27.539185 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.539149    5005 generic.go:356] "Generic (PLEG): container finished" podID="00145046-700c-436b-a729-4332d8935371" containerID="24ec2c025816d2a18e1595dfd26e3cb3a74ab4e2ca6e8b525ba9131025414df0" exitCode=0
Nov 03 02:22:27.539366 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.539226    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl" event={"ID":"00145046-700c-436b-a729-4332d8935371","Type":"ContainerDied","Data":"24ec2c025816d2a18e1595dfd26e3cb3a74ab4e2ca6e8b525ba9131025414df0"}
Nov 03 02:22:27.540649 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.540624    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk" event={"ID":"7ebd4e3a-53a6-414a-94a2-726e9815b7aa","Type":"ContainerStarted","Data":"c2d5af2cf99f0bf0a9c915364e5a4f22b78c013227d939823981b2350057dfb1"}
Nov 03 02:22:27.650836 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.650775    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:22:27.724923 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.724883    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-workload-certs\") pod \"00145046-700c-436b-a729-4332d8935371\" (UID: \"00145046-700c-436b-a729-4332d8935371\") "
Nov 03 02:22:27.725088 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.724975    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-credential-socket\") pod \"00145046-700c-436b-a729-4332d8935371\" (UID: \"00145046-700c-436b-a729-4332d8935371\") "
Nov 03 02:22:27.725150 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.725109    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00145046-700c-436b-a729-4332d8935371-istio-token\" (UniqueName: \"kubernetes.io/projected/00145046-700c-436b-a729-4332d8935371-istio-token\") pod \"00145046-700c-436b-a729-4332d8935371\" (UID: \"00145046-700c-436b-a729-4332d8935371\") "
Nov 03 02:22:27.725150 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.725136    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-workload-socket\") pod \"00145046-700c-436b-a729-4332d8935371\" (UID: \"00145046-700c-436b-a729-4332d8935371\") "
Nov 03 02:22:27.725277 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.725172    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00145046-700c-436b-a729-4332d8935371-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00145046-700c-436b-a729-4332d8935371-istio-podinfo\") pod \"00145046-700c-436b-a729-4332d8935371\" (UID: \"00145046-700c-436b-a729-4332d8935371\") "
Nov 03 02:22:27.725277 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.725235    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-istio-envoy\") pod \"00145046-700c-436b-a729-4332d8935371\" (UID: \"00145046-700c-436b-a729-4332d8935371\") "
Nov 03 02:22:27.725277 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.725256    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-workload-certs" pod "00145046-700c-436b-a729-4332d8935371" (UID: "00145046-700c-436b-a729-4332d8935371"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:22:27.725433 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.725329    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00145046-700c-436b-a729-4332d8935371-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00145046-700c-436b-a729-4332d8935371-istiod-ca-cert\") pod \"00145046-700c-436b-a729-4332d8935371\" (UID: \"00145046-700c-436b-a729-4332d8935371\") "
Nov 03 02:22:27.725433 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.725366    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-credential-socket" pod "00145046-700c-436b-a729-4332d8935371" (UID: "00145046-700c-436b-a729-4332d8935371"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:22:27.725433 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.725363    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00145046-700c-436b-a729-4332d8935371-kube-api-access-mrzrz\" (UniqueName: \"kubernetes.io/projected/00145046-700c-436b-a729-4332d8935371-kube-api-access-mrzrz\") pod \"00145046-700c-436b-a729-4332d8935371\" (UID: \"00145046-700c-436b-a729-4332d8935371\") "
Nov 03 02:22:27.725590 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.725447    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-istio-data\") pod \"00145046-700c-436b-a729-4332d8935371\" (UID: \"00145046-700c-436b-a729-4332d8935371\") "
Nov 03 02:22:27.725709 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.725670    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-workload-socket" pod "00145046-700c-436b-a729-4332d8935371" (UID: "00145046-700c-436b-a729-4332d8935371"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:22:27.725780 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.725684    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00145046-700c-436b-a729-4332d8935371-istiod-ca-cert" pod "00145046-700c-436b-a729-4332d8935371" (UID: "00145046-700c-436b-a729-4332d8935371"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:22:27.725978 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.725955    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-istio-data" pod "00145046-700c-436b-a729-4332d8935371" (UID: "00145046-700c-436b-a729-4332d8935371"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:22:27.726141 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.726110    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:27.726141 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.726127    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00145046-700c-436b-a729-4332d8935371-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:27.726141 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.726142    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:27.726390 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.726155    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:27.726390 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.726167    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:27.727805 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.727782    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-istio-envoy" pod "00145046-700c-436b-a729-4332d8935371" (UID: "00145046-700c-436b-a729-4332d8935371"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:22:27.727914 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.727830    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00145046-700c-436b-a729-4332d8935371-istio-token" pod "00145046-700c-436b-a729-4332d8935371" (UID: "00145046-700c-436b-a729-4332d8935371"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:22:27.727977 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.727936    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00145046-700c-436b-a729-4332d8935371-istio-podinfo" pod "00145046-700c-436b-a729-4332d8935371" (UID: "00145046-700c-436b-a729-4332d8935371"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:22:27.728104 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.728083    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00145046-700c-436b-a729-4332d8935371-kube-api-access-mrzrz" pod "00145046-700c-436b-a729-4332d8935371" (UID: "00145046-700c-436b-a729-4332d8935371"). InnerVolumeSpecName "kube-api-access-mrzrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:22:27.828326 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.828288    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00145046-700c-436b-a729-4332d8935371-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:27.828326 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.828321    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00145046-700c-436b-a729-4332d8935371-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:27.828326 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.828332    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00145046-700c-436b-a729-4332d8935371-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:27.828838 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:27.828342    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrzrz\" (UniqueName: \"kubernetes.io/projected/00145046-700c-436b-a729-4332d8935371-kube-api-access-mrzrz\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:28.548969 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:28.548931    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl" event={"ID":"00145046-700c-436b-a729-4332d8935371","Type":"ContainerDied","Data":"f03acb1ecd57fb3bb64af461b0c869770a97aebaed76d77f93d9bc470f70f00c"}
Nov 03 02:22:28.549168 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:28.548943    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"
Nov 03 02:22:28.549168 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:28.549000    5005 scope.go:122] "RemoveContainer" containerID="24ec2c025816d2a18e1595dfd26e3cb3a74ab4e2ca6e8b525ba9131025414df0"
Nov 03 02:22:28.550923 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:28.550900    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk" event={"ID":"7ebd4e3a-53a6-414a-94a2-726e9815b7aa","Type":"ContainerStarted","Data":"35b6cfe9faa938876ebc025a9d9064753382b0f4665ef11fd0728bbff68f2c23"}
Nov 03 02:22:28.551070 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:28.551048    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"
Nov 03 02:22:28.705166 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:28.705130    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"]
Nov 03 02:22:28.749941 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:28.749909    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-vwrnl"]
Nov 03 02:22:28.815773 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:28.815662    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk" podStartSLOduration=2.975753515 podStartE2EDuration="3.815645376s" totalImagesPullingTime="839.891861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:22:28.812784808 +0000 UTC m=+1875.827026173" watchObservedRunningTime="2026-11-03 02:22:28.815645376 +0000 UTC m=+1875.829886743"
Nov 03 02:22:29.113596 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:29.113523    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00145046-700c-436b-a729-4332d8935371" path="/var/lib/kubelet/pods/00145046-700c-436b-a729-4332d8935371/volumes"
Nov 03 02:22:30.714560 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:22:30.714522    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:22:34.565564 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:22:34.565534    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"
Nov 03 02:23:00.715534 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:23:00.715491    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:23:18.032956 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:18.032909    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"]
Nov 03 02:23:18.033435 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:18.033135    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk" podUID="bf895979-b958-4260-8a57-9ca10a4501d7" containerName="istio-proxy" containerID="cri-o://d525383963c354e8cfca91500d10808330313992c3206ea2f0a46cb489c5b3bf" gracePeriod=30
Nov 03 02:23:21.885400 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:21.885365    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 02:23:21.885836 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:21.885430    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk" podUID="bf895979-b958-4260-8a57-9ca10a4501d7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 02:23:23.301996 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.301932    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:23:23.397660 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.397625    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-credential-socket\") pod \"bf895979-b958-4260-8a57-9ca10a4501d7\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") "
Nov 03 02:23:23.397849 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.397671    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf895979-b958-4260-8a57-9ca10a4501d7-istio-token\" (UniqueName: \"kubernetes.io/projected/bf895979-b958-4260-8a57-9ca10a4501d7-istio-token\") pod \"bf895979-b958-4260-8a57-9ca10a4501d7\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") "
Nov 03 02:23:23.397849 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.397748    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-istio-envoy\") pod \"bf895979-b958-4260-8a57-9ca10a4501d7\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") "
Nov 03 02:23:23.397849 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.397795    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf895979-b958-4260-8a57-9ca10a4501d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf895979-b958-4260-8a57-9ca10a4501d7-istio-podinfo\") pod \"bf895979-b958-4260-8a57-9ca10a4501d7\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") "
Nov 03 02:23:23.397996 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.397847    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf895979-b958-4260-8a57-9ca10a4501d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf895979-b958-4260-8a57-9ca10a4501d7-istiod-ca-cert\") pod \"bf895979-b958-4260-8a57-9ca10a4501d7\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") "
Nov 03 02:23:23.397996 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.397951    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-workload-socket\") pod \"bf895979-b958-4260-8a57-9ca10a4501d7\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") "
Nov 03 02:23:23.397996 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.397972    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-credential-socket" pod "bf895979-b958-4260-8a57-9ca10a4501d7" (UID: "bf895979-b958-4260-8a57-9ca10a4501d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:23:23.398149 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.397995    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf895979-b958-4260-8a57-9ca10a4501d7-kube-api-access-hv969\" (UniqueName: \"kubernetes.io/projected/bf895979-b958-4260-8a57-9ca10a4501d7-kube-api-access-hv969\") pod \"bf895979-b958-4260-8a57-9ca10a4501d7\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") "
Nov 03 02:23:23.398149 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.398077    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-istio-data\") pod \"bf895979-b958-4260-8a57-9ca10a4501d7\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") "
Nov 03 02:23:23.398149 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.398115    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-workload-certs\") pod \"bf895979-b958-4260-8a57-9ca10a4501d7\" (UID: \"bf895979-b958-4260-8a57-9ca10a4501d7\") "
Nov 03 02:23:23.398309 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.398215    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-workload-socket" pod "bf895979-b958-4260-8a57-9ca10a4501d7" (UID: "bf895979-b958-4260-8a57-9ca10a4501d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:23:23.398309 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.398285    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf895979-b958-4260-8a57-9ca10a4501d7-istiod-ca-cert" pod "bf895979-b958-4260-8a57-9ca10a4501d7" (UID: "bf895979-b958-4260-8a57-9ca10a4501d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:23:23.398460 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.398424    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-istio-data" pod "bf895979-b958-4260-8a57-9ca10a4501d7" (UID: "bf895979-b958-4260-8a57-9ca10a4501d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:23:23.398520 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.398493    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-workload-certs" pod "bf895979-b958-4260-8a57-9ca10a4501d7" (UID: "bf895979-b958-4260-8a57-9ca10a4501d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:23:23.398746 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.398727    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:23.398805 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.398753    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:23.398805 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.398767    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:23.398805 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.398782    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:23.398805 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.398799    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf895979-b958-4260-8a57-9ca10a4501d7-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:23.400114 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.400088    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf895979-b958-4260-8a57-9ca10a4501d7-istio-token" pod "bf895979-b958-4260-8a57-9ca10a4501d7" (UID: "bf895979-b958-4260-8a57-9ca10a4501d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:23:23.400228 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.400121    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-istio-envoy" pod "bf895979-b958-4260-8a57-9ca10a4501d7" (UID: "bf895979-b958-4260-8a57-9ca10a4501d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:23:23.400434 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.400415    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf895979-b958-4260-8a57-9ca10a4501d7-kube-api-access-hv969" pod "bf895979-b958-4260-8a57-9ca10a4501d7" (UID: "bf895979-b958-4260-8a57-9ca10a4501d7"). InnerVolumeSpecName "kube-api-access-hv969". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:23:23.400481 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.400418    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf895979-b958-4260-8a57-9ca10a4501d7-istio-podinfo" pod "bf895979-b958-4260-8a57-9ca10a4501d7" (UID: "bf895979-b958-4260-8a57-9ca10a4501d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:23:23.500637 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.500595    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv969\" (UniqueName: \"kubernetes.io/projected/bf895979-b958-4260-8a57-9ca10a4501d7-kube-api-access-hv969\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:23.500637 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.500627    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf895979-b958-4260-8a57-9ca10a4501d7-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:23.500637 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.500638    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf895979-b958-4260-8a57-9ca10a4501d7-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:23.500970 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.500649    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf895979-b958-4260-8a57-9ca10a4501d7-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:23.947383 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.947343    5005 generic.go:356] "Generic (PLEG): container finished" podID="bf895979-b958-4260-8a57-9ca10a4501d7" containerID="d525383963c354e8cfca91500d10808330313992c3206ea2f0a46cb489c5b3bf" exitCode=0
Nov 03 02:23:23.947606 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.947400    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk" event={"ID":"bf895979-b958-4260-8a57-9ca10a4501d7","Type":"ContainerDied","Data":"d525383963c354e8cfca91500d10808330313992c3206ea2f0a46cb489c5b3bf"}
Nov 03 02:23:23.947606 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.947402    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"
Nov 03 02:23:23.947606 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.947446    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk" event={"ID":"bf895979-b958-4260-8a57-9ca10a4501d7","Type":"ContainerDied","Data":"d8c98ae08379e1d356fa29eeb16b842ca2cdb9c499abcc6eaf1e013ef997020f"}
Nov 03 02:23:23.947606 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.947473    5005 scope.go:122] "RemoveContainer" containerID="d525383963c354e8cfca91500d10808330313992c3206ea2f0a46cb489c5b3bf"
Nov 03 02:23:23.958600 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.958581    5005 scope.go:122] "RemoveContainer" containerID="d525383963c354e8cfca91500d10808330313992c3206ea2f0a46cb489c5b3bf"
Nov 03 02:23:23.958900 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:23:23.958878    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d525383963c354e8cfca91500d10808330313992c3206ea2f0a46cb489c5b3bf\": container with ID starting with d525383963c354e8cfca91500d10808330313992c3206ea2f0a46cb489c5b3bf not found: ID does not exist" containerID="d525383963c354e8cfca91500d10808330313992c3206ea2f0a46cb489c5b3bf"
Nov 03 02:23:23.958967 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:23.958911    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d525383963c354e8cfca91500d10808330313992c3206ea2f0a46cb489c5b3bf"} err="failed to get container status \"d525383963c354e8cfca91500d10808330313992c3206ea2f0a46cb489c5b3bf\": rpc error: code = NotFound desc = could not find container \"d525383963c354e8cfca91500d10808330313992c3206ea2f0a46cb489c5b3bf\": container with ID starting with d525383963c354e8cfca91500d10808330313992c3206ea2f0a46cb489c5b3bf not found: ID does not exist"
Nov 03 02:23:24.102583 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:24.102548    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"]
Nov 03 02:23:24.153879 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:24.153824    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-wbltk"]
Nov 03 02:23:25.109406 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:23:25.109369    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf895979-b958-4260-8a57-9ca10a4501d7" path="/var/lib/kubelet/pods/bf895979-b958-4260-8a57-9ca10a4501d7/volumes"
Nov 03 02:23:30.715476 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:23:30.715432    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:24:00.717009 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:24:00.716964    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:24:28.555736 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:28.555688    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"]
Nov 03 02:24:28.556293 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:28.556278    5005 cpu_manager.go:414] "containerMap: removing container" podUID="bf895979-b958-4260-8a57-9ca10a4501d7" containerName="istio-proxy"
Nov 03 02:24:28.556391 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:28.556375    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf895979-b958-4260-8a57-9ca10a4501d7" containerName="istio-proxy"
Nov 03 02:24:28.556450 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:28.556409    5005 cpu_manager.go:414] "containerMap: removing container" podUID="00145046-700c-436b-a729-4332d8935371" containerName="istio-proxy"
Nov 03 02:24:28.556450 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:28.556415    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00145046-700c-436b-a729-4332d8935371" containerName="istio-proxy"
Nov 03 02:24:28.556577 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:28.556567    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="00145046-700c-436b-a729-4332d8935371" containerName="istio-proxy"
Nov 03 02:24:28.556618 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:28.556587    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf895979-b958-4260-8a57-9ca10a4501d7" containerName="istio-proxy"
Nov 03 02:24:28.561461 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:28.561440    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"
Nov 03 02:24:28.601215 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:28.601178    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-metrics-certs\") pod \"maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6\" (UID: \"ffe1671a-c31c-4dc0-85a2-51d0b97daa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"
Nov 03 02:24:28.601434 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:28.601257    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6\" (UID: \"ffe1671a-c31c-4dc0-85a2-51d0b97daa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"
Nov 03 02:24:28.601496 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:28.601457    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf8jd\" (UniqueName: \"kubernetes.io/projected/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-kube-api-access-qf8jd\") pod \"maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6\" (UID: \"ffe1671a-c31c-4dc0-85a2-51d0b97daa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"
Nov 03 02:24:28.601580 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:28.601557    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6\" (UID: \"ffe1671a-c31c-4dc0-85a2-51d0b97daa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"
Nov 03 02:24:28.613621 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:28.613594    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"]
Nov 03 02:24:28.655380 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:28.655342    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-c62c3d-serving-cert\""
Nov 03 02:24:28.703615 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:28.703578    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf8jd\" (UniqueName: \"kubernetes.io/projected/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-kube-api-access-qf8jd\") pod \"maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6\" (UID: \"ffe1671a-c31c-4dc0-85a2-51d0b97daa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"
Nov 03 02:24:28.703832 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:28.703652    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6\" (UID: \"ffe1671a-c31c-4dc0-85a2-51d0b97daa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"
Nov 03 02:24:28.704588 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:28.704281    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-metrics-certs\") pod \"maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6\" (UID: \"ffe1671a-c31c-4dc0-85a2-51d0b97daa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"
Nov 03 02:24:28.704588 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:28.704388    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6\" (UID: \"ffe1671a-c31c-4dc0-85a2-51d0b97daa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"
Nov 03 02:24:28.704588 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:28.704427    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6\" (UID: \"ffe1671a-c31c-4dc0-85a2-51d0b97daa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"
Nov 03 02:24:28.705175 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:24:28.704907    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-serving-cert: secret "maas-api-e2e-praxis-a-c62c3d-serving-cert" not found
Nov 03 02:24:28.705175 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:24:28.704972    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-maas-api-tls podName:ffe1671a-c31c-4dc0-85a2-51d0b97daa81 nodeName:}" failed. No retries permitted until 2026-11-03 02:24:29.204952144 +0000 UTC m=+1996.219193492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-maas-api-tls") pod "maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6" (UID: "ffe1671a-c31c-4dc0-85a2-51d0b97daa81") : secret "maas-api-e2e-praxis-a-c62c3d-serving-cert" not found
Nov 03 02:24:28.709338 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:28.709318    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-metrics-certs\") pod \"maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6\" (UID: \"ffe1671a-c31c-4dc0-85a2-51d0b97daa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"
Nov 03 02:24:28.757253 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:28.757217    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf8jd\" (UniqueName: \"kubernetes.io/projected/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-kube-api-access-qf8jd\") pod \"maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6\" (UID: \"ffe1671a-c31c-4dc0-85a2-51d0b97daa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"
Nov 03 02:24:29.222757 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:29.222685    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6\" (UID: \"ffe1671a-c31c-4dc0-85a2-51d0b97daa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"
Nov 03 02:24:29.225223 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:29.225197    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6\" (UID: \"ffe1671a-c31c-4dc0-85a2-51d0b97daa81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"
Nov 03 02:24:29.475592 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:29.475496    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"
Nov 03 02:24:29.850212 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:29.850173    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"]
Nov 03 02:24:29.859301 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:24:29.859270    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe1671a_c31c_4dc0_85a2_51d0b97daa81.slice/crio-69f2149878b5d5b822e0a033a7a0df4043dd60a1e02d1cf8433fdc091f9954f5 WatchSource:0}: Error finding container 69f2149878b5d5b822e0a033a7a0df4043dd60a1e02d1cf8433fdc091f9954f5: Status 404 returned error can't find the container with id 69f2149878b5d5b822e0a033a7a0df4043dd60a1e02d1cf8433fdc091f9954f5
Nov 03 02:24:30.431850 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:30.431806    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6" event={"ID":"ffe1671a-c31c-4dc0-85a2-51d0b97daa81","Type":"ContainerStarted","Data":"69f2149878b5d5b822e0a033a7a0df4043dd60a1e02d1cf8433fdc091f9954f5"}
Nov 03 02:24:30.725788 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:24:30.725750    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:24:31.439085 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:31.439041    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6" event={"ID":"ffe1671a-c31c-4dc0-85a2-51d0b97daa81","Type":"ContainerStarted","Data":"ef140c4025083d0c33e88cf8279d1bbd3106363ee44a7425e96aaf275756bff5"}
Nov 03 02:24:31.439497 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:31.439140    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"
Nov 03 02:24:31.546158 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:31.546094    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6" podStartSLOduration=2.742872224 podStartE2EDuration="3.546079373s" totalImagesPullingTime="803.207149ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:24:31.545862927 +0000 UTC m=+1998.560104294" watchObservedRunningTime="2026-11-03 02:24:31.546079373 +0000 UTC m=+1998.560320740"
Nov 03 02:24:37.451214 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:24:37.451178    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"
Nov 03 02:25:00.716616 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:25:00.716534    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:25:05.579201 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:05.579170    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"]
Nov 03 02:25:05.579545 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:05.579351    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8" podUID="1ee5f1b4-095d-42e3-84be-3a8b2e4400ee" containerName="maas-api" containerID="cri-o://0cad936a8ba0266e0c7a002ca87734ccb7d0dd55b3fbddafcba63f13366bcb6f" gracePeriod=30
Nov 03 02:25:05.889334 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:05.889305    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"
Nov 03 02:25:05.915526 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:05.915491    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-kube-api-access-t9gzt\" (UniqueName: \"kubernetes.io/projected/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-kube-api-access-t9gzt\") pod \"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee\" (UID: \"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee\") "
Nov 03 02:25:05.915779 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:05.915587    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-maas-api-tls\") pod \"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee\" (UID: \"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee\") "
Nov 03 02:25:05.915779 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:05.915725    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-metrics-certs\") pod \"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee\" (UID: \"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee\") "
Nov 03 02:25:05.915940 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:05.915851    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-openshift-service-ca\") pod \"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee\" (UID: \"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee\") "
Nov 03 02:25:05.916235 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:05.916206    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-openshift-service-ca" pod "1ee5f1b4-095d-42e3-84be-3a8b2e4400ee" (UID: "1ee5f1b4-095d-42e3-84be-3a8b2e4400ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:25:05.916466 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:05.916421    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:25:05.917818 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:05.917776    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-maas-api-tls" pod "1ee5f1b4-095d-42e3-84be-3a8b2e4400ee" (UID: "1ee5f1b4-095d-42e3-84be-3a8b2e4400ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:25:05.917922 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:05.917838    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-metrics-certs" pod "1ee5f1b4-095d-42e3-84be-3a8b2e4400ee" (UID: "1ee5f1b4-095d-42e3-84be-3a8b2e4400ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:25:05.917922 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:05.917861    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-kube-api-access-t9gzt" pod "1ee5f1b4-095d-42e3-84be-3a8b2e4400ee" (UID: "1ee5f1b4-095d-42e3-84be-3a8b2e4400ee"). InnerVolumeSpecName "kube-api-access-t9gzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:25:06.018290 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:06.018256    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9gzt\" (UniqueName: \"kubernetes.io/projected/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-kube-api-access-t9gzt\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:25:06.018290 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:06.018290    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:25:06.018491 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:06.018304    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:25:06.747760 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:06.743228    5005 generic.go:356] "Generic (PLEG): container finished" podID="1ee5f1b4-095d-42e3-84be-3a8b2e4400ee" containerID="0cad936a8ba0266e0c7a002ca87734ccb7d0dd55b3fbddafcba63f13366bcb6f" exitCode=0
Nov 03 02:25:06.747760 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:06.743492    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8" event={"ID":"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee","Type":"ContainerDied","Data":"0cad936a8ba0266e0c7a002ca87734ccb7d0dd55b3fbddafcba63f13366bcb6f"}
Nov 03 02:25:06.747760 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:06.743633    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8" event={"ID":"1ee5f1b4-095d-42e3-84be-3a8b2e4400ee","Type":"ContainerDied","Data":"3e4714fe774c453a01cc1eb88437ee39600297784c47467015c26a080f788c8b"}
Nov 03 02:25:06.747760 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:06.743712    5005 scope.go:122] "RemoveContainer" containerID="0cad936a8ba0266e0c7a002ca87734ccb7d0dd55b3fbddafcba63f13366bcb6f"
Nov 03 02:25:06.747760 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:06.743986    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"
Nov 03 02:25:06.761768 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:06.761746    5005 scope.go:122] "RemoveContainer" containerID="0cad936a8ba0266e0c7a002ca87734ccb7d0dd55b3fbddafcba63f13366bcb6f"
Nov 03 02:25:06.762149 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:25:06.762123    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cad936a8ba0266e0c7a002ca87734ccb7d0dd55b3fbddafcba63f13366bcb6f\": container with ID starting with 0cad936a8ba0266e0c7a002ca87734ccb7d0dd55b3fbddafcba63f13366bcb6f not found: ID does not exist" containerID="0cad936a8ba0266e0c7a002ca87734ccb7d0dd55b3fbddafcba63f13366bcb6f"
Nov 03 02:25:06.762219 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:06.762162    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cad936a8ba0266e0c7a002ca87734ccb7d0dd55b3fbddafcba63f13366bcb6f"} err="failed to get container status \"0cad936a8ba0266e0c7a002ca87734ccb7d0dd55b3fbddafcba63f13366bcb6f\": rpc error: code = NotFound desc = could not find container \"0cad936a8ba0266e0c7a002ca87734ccb7d0dd55b3fbddafcba63f13366bcb6f\": container with ID starting with 0cad936a8ba0266e0c7a002ca87734ccb7d0dd55b3fbddafcba63f13366bcb6f not found: ID does not exist"
Nov 03 02:25:06.903154 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:06.903119    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"]
Nov 03 02:25:06.908353 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:06.908325    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f66575-57cf57b54d-5vhz8"]
Nov 03 02:25:07.109956 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:07.109913    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ee5f1b4-095d-42e3-84be-3a8b2e4400ee" path="/var/lib/kubelet/pods/1ee5f1b4-095d-42e3-84be-3a8b2e4400ee/volumes"
Nov 03 02:25:07.292883 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:07.292845    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"]
Nov 03 02:25:07.293619 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:07.293593    5005 cpu_manager.go:414] "containerMap: removing container" podUID="1ee5f1b4-095d-42e3-84be-3a8b2e4400ee" containerName="maas-api"
Nov 03 02:25:07.293619 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:07.293617    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ee5f1b4-095d-42e3-84be-3a8b2e4400ee" containerName="maas-api"
Nov 03 02:25:07.293880 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:07.293781    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ee5f1b4-095d-42e3-84be-3a8b2e4400ee" containerName="maas-api"
Nov 03 02:25:07.298640 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:07.298619    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"
Nov 03 02:25:07.343470 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:07.343434    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7px79\" (UniqueName: \"kubernetes.io/projected/c99ef35d-16a1-435b-afa3-203d04df4f1c-kube-api-access-7px79\") pod \"maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc\" (UID: \"c99ef35d-16a1-435b-afa3-203d04df4f1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"
Nov 03 02:25:07.343633 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:07.343481    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c99ef35d-16a1-435b-afa3-203d04df4f1c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc\" (UID: \"c99ef35d-16a1-435b-afa3-203d04df4f1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"
Nov 03 02:25:07.343633 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:07.343575    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c99ef35d-16a1-435b-afa3-203d04df4f1c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc\" (UID: \"c99ef35d-16a1-435b-afa3-203d04df4f1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"
Nov 03 02:25:07.343744 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:07.343713    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99ef35d-16a1-435b-afa3-203d04df4f1c-metrics-certs\") pod \"maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc\" (UID: \"c99ef35d-16a1-435b-afa3-203d04df4f1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"
Nov 03 02:25:07.347580 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:07.347555    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-8857cb-serving-cert\""
Nov 03 02:25:07.350959 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:07.350931    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"]
Nov 03 02:25:07.444999 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:07.444910    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7px79\" (UniqueName: \"kubernetes.io/projected/c99ef35d-16a1-435b-afa3-203d04df4f1c-kube-api-access-7px79\") pod \"maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc\" (UID: \"c99ef35d-16a1-435b-afa3-203d04df4f1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"
Nov 03 02:25:07.444999 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:07.444957    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c99ef35d-16a1-435b-afa3-203d04df4f1c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc\" (UID: \"c99ef35d-16a1-435b-afa3-203d04df4f1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"
Nov 03 02:25:07.444999 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:07.444997    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c99ef35d-16a1-435b-afa3-203d04df4f1c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc\" (UID: \"c99ef35d-16a1-435b-afa3-203d04df4f1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"
Nov 03 02:25:07.445282 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:07.445118    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99ef35d-16a1-435b-afa3-203d04df4f1c-metrics-certs\") pod \"maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc\" (UID: \"c99ef35d-16a1-435b-afa3-203d04df4f1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"
Nov 03 02:25:07.445282 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:25:07.445120    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-serving-cert: secret "maas-api-e2e-ipp-b-8857cb-serving-cert" not found
Nov 03 02:25:07.445282 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:25:07.445270    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c99ef35d-16a1-435b-afa3-203d04df4f1c-maas-api-tls podName:c99ef35d-16a1-435b-afa3-203d04df4f1c nodeName:}" failed. No retries permitted until 2026-11-03 02:25:07.945253286 +0000 UTC m=+2034.959494640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c99ef35d-16a1-435b-afa3-203d04df4f1c-maas-api-tls") pod "maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc" (UID: "c99ef35d-16a1-435b-afa3-203d04df4f1c") : secret "maas-api-e2e-ipp-b-8857cb-serving-cert" not found
Nov 03 02:25:07.445850 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:07.445831    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c99ef35d-16a1-435b-afa3-203d04df4f1c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc\" (UID: \"c99ef35d-16a1-435b-afa3-203d04df4f1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"
Nov 03 02:25:07.447591 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:07.447567    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99ef35d-16a1-435b-afa3-203d04df4f1c-metrics-certs\") pod \"maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc\" (UID: \"c99ef35d-16a1-435b-afa3-203d04df4f1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"
Nov 03 02:25:07.498566 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:07.498532    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7px79\" (UniqueName: \"kubernetes.io/projected/c99ef35d-16a1-435b-afa3-203d04df4f1c-kube-api-access-7px79\") pod \"maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc\" (UID: \"c99ef35d-16a1-435b-afa3-203d04df4f1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"
Nov 03 02:25:07.952490 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:07.952453    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c99ef35d-16a1-435b-afa3-203d04df4f1c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc\" (UID: \"c99ef35d-16a1-435b-afa3-203d04df4f1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"
Nov 03 02:25:07.955175 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:07.955154    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c99ef35d-16a1-435b-afa3-203d04df4f1c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc\" (UID: \"c99ef35d-16a1-435b-afa3-203d04df4f1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"
Nov 03 02:25:08.212289 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:08.212196    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"
Nov 03 02:25:08.589275 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:08.589244    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"]
Nov 03 02:25:08.594381 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:25:08.594355    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc99ef35d_16a1_435b_afa3_203d04df4f1c.slice/crio-0b4e4968a2aad4d52b00ef590526cb6f694eae008615bede1b24c848ece5fcc2 WatchSource:0}: Error finding container 0b4e4968a2aad4d52b00ef590526cb6f694eae008615bede1b24c848ece5fcc2: Status 404 returned error can't find the container with id 0b4e4968a2aad4d52b00ef590526cb6f694eae008615bede1b24c848ece5fcc2
Nov 03 02:25:08.595921 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:08.595899    5005 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 02:25:08.773654 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:08.773615    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc" event={"ID":"c99ef35d-16a1-435b-afa3-203d04df4f1c","Type":"ContainerStarted","Data":"0b4e4968a2aad4d52b00ef590526cb6f694eae008615bede1b24c848ece5fcc2"}
Nov 03 02:25:09.782132 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:09.782096    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc" event={"ID":"c99ef35d-16a1-435b-afa3-203d04df4f1c","Type":"ContainerStarted","Data":"b8f8701d5c0e3d005d724ccece3f6c1b33a08489dd167cc7e410a56d119f079d"}
Nov 03 02:25:09.782576 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:09.782161    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"
Nov 03 02:25:09.890612 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:09.890517    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc" podStartSLOduration=1.908193395 podStartE2EDuration="2.89050235s" totalImagesPullingTime="982.308955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:25:09.888232209 +0000 UTC m=+2036.902473575" watchObservedRunningTime="2026-11-03 02:25:09.89050235 +0000 UTC m=+2036.904743716"
Nov 03 02:25:13.188636 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:13.188602    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"]
Nov 03 02:25:13.189171 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:13.188805    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz" podUID="af839e9e-267b-4c27-b8cc-4e4e591580e3" containerName="istio-proxy" containerID="cri-o://2d701cb3cf9e41322d8b8d01265d13fdc81ef21fa27aa6d2f3423d7c0e6643df" gracePeriod=30
Nov 03 02:25:14.697980 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:14.697941    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-h4htz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 02:25:14.698431 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:14.698017    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz" podUID="af839e9e-267b-4c27-b8cc-4e4e591580e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 02:25:15.795494 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:15.795456    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"
Nov 03 02:25:18.456015 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.455986    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:25:18.601074 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.601041    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-envoy\") pod \"af839e9e-267b-4c27-b8cc-4e4e591580e3\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") "
Nov 03 02:25:18.601246 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.601159    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-token\" (UniqueName: \"kubernetes.io/projected/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-token\") pod \"af839e9e-267b-4c27-b8cc-4e4e591580e3\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") "
Nov 03 02:25:18.601246 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.601199    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-data\") pod \"af839e9e-267b-4c27-b8cc-4e4e591580e3\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") "
Nov 03 02:25:18.601333 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.601258    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af839e9e-267b-4c27-b8cc-4e4e591580e3-kube-api-access-t9m2r\" (UniqueName: \"kubernetes.io/projected/af839e9e-267b-4c27-b8cc-4e4e591580e3-kube-api-access-t9m2r\") pod \"af839e9e-267b-4c27-b8cc-4e4e591580e3\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") "
Nov 03 02:25:18.601333 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.601305    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af839e9e-267b-4c27-b8cc-4e4e591580e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af839e9e-267b-4c27-b8cc-4e4e591580e3-istiod-ca-cert\") pod \"af839e9e-267b-4c27-b8cc-4e4e591580e3\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") "
Nov 03 02:25:18.601414 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.601344    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-credential-socket\") pod \"af839e9e-267b-4c27-b8cc-4e4e591580e3\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") "
Nov 03 02:25:18.601476 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.601450    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-workload-socket\") pod \"af839e9e-267b-4c27-b8cc-4e4e591580e3\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") "
Nov 03 02:25:18.601531 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.601491    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-workload-certs\") pod \"af839e9e-267b-4c27-b8cc-4e4e591580e3\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") "
Nov 03 02:25:18.601581 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.601552    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-podinfo\") pod \"af839e9e-267b-4c27-b8cc-4e4e591580e3\" (UID: \"af839e9e-267b-4c27-b8cc-4e4e591580e3\") "
Nov 03 02:25:18.601630 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.601573    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-data" pod "af839e9e-267b-4c27-b8cc-4e4e591580e3" (UID: "af839e9e-267b-4c27-b8cc-4e4e591580e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:25:18.601817 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.601737    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-workload-socket" pod "af839e9e-267b-4c27-b8cc-4e4e591580e3" (UID: "af839e9e-267b-4c27-b8cc-4e4e591580e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:25:18.601907 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.601812    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-workload-certs" pod "af839e9e-267b-4c27-b8cc-4e4e591580e3" (UID: "af839e9e-267b-4c27-b8cc-4e4e591580e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:25:18.601970 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.601920    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-credential-socket" pod "af839e9e-267b-4c27-b8cc-4e4e591580e3" (UID: "af839e9e-267b-4c27-b8cc-4e4e591580e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:25:18.602074 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.602050    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af839e9e-267b-4c27-b8cc-4e4e591580e3-istiod-ca-cert" pod "af839e9e-267b-4c27-b8cc-4e4e591580e3" (UID: "af839e9e-267b-4c27-b8cc-4e4e591580e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:25:18.602204 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.602188    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af839e9e-267b-4c27-b8cc-4e4e591580e3-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:25:18.602271 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.602207    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:25:18.602271 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.602216    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:25:18.602271 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.602227    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:25:18.602271 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.602241    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:25:18.603421 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.603404    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-envoy" pod "af839e9e-267b-4c27-b8cc-4e4e591580e3" (UID: "af839e9e-267b-4c27-b8cc-4e4e591580e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:25:18.603742 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.603723    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-podinfo" pod "af839e9e-267b-4c27-b8cc-4e4e591580e3" (UID: "af839e9e-267b-4c27-b8cc-4e4e591580e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:25:18.603900 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.603879    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-token" pod "af839e9e-267b-4c27-b8cc-4e4e591580e3" (UID: "af839e9e-267b-4c27-b8cc-4e4e591580e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:25:18.603942 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.603912    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af839e9e-267b-4c27-b8cc-4e4e591580e3-kube-api-access-t9m2r" pod "af839e9e-267b-4c27-b8cc-4e4e591580e3" (UID: "af839e9e-267b-4c27-b8cc-4e4e591580e3"). InnerVolumeSpecName "kube-api-access-t9m2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:25:18.703351 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.703316    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:25:18.703351 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.703349    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9m2r\" (UniqueName: \"kubernetes.io/projected/af839e9e-267b-4c27-b8cc-4e4e591580e3-kube-api-access-t9m2r\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:25:18.703557 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.703364    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:25:18.703557 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.703376    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af839e9e-267b-4c27-b8cc-4e4e591580e3-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:25:18.846611 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.846573    5005 generic.go:356] "Generic (PLEG): container finished" podID="af839e9e-267b-4c27-b8cc-4e4e591580e3" containerID="2d701cb3cf9e41322d8b8d01265d13fdc81ef21fa27aa6d2f3423d7c0e6643df" exitCode=0
Nov 03 02:25:18.846808 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.846619    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"
Nov 03 02:25:18.846808 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.846660    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz" event={"ID":"af839e9e-267b-4c27-b8cc-4e4e591580e3","Type":"ContainerDied","Data":"2d701cb3cf9e41322d8b8d01265d13fdc81ef21fa27aa6d2f3423d7c0e6643df"}
Nov 03 02:25:18.846808 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.846728    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz" event={"ID":"af839e9e-267b-4c27-b8cc-4e4e591580e3","Type":"ContainerDied","Data":"ce6e7bbacdf87a5eef93bc6a9c85af098498eb3fccdcdccbab21e9dcc1c63c2f"}
Nov 03 02:25:18.846808 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.846756    5005 scope.go:122] "RemoveContainer" containerID="2d701cb3cf9e41322d8b8d01265d13fdc81ef21fa27aa6d2f3423d7c0e6643df"
Nov 03 02:25:18.858970 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.858952    5005 scope.go:122] "RemoveContainer" containerID="2d701cb3cf9e41322d8b8d01265d13fdc81ef21fa27aa6d2f3423d7c0e6643df"
Nov 03 02:25:18.859235 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:25:18.859214    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d701cb3cf9e41322d8b8d01265d13fdc81ef21fa27aa6d2f3423d7c0e6643df\": container with ID starting with 2d701cb3cf9e41322d8b8d01265d13fdc81ef21fa27aa6d2f3423d7c0e6643df not found: ID does not exist" containerID="2d701cb3cf9e41322d8b8d01265d13fdc81ef21fa27aa6d2f3423d7c0e6643df"
Nov 03 02:25:18.859289 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:18.859244    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d701cb3cf9e41322d8b8d01265d13fdc81ef21fa27aa6d2f3423d7c0e6643df"} err="failed to get container status \"2d701cb3cf9e41322d8b8d01265d13fdc81ef21fa27aa6d2f3423d7c0e6643df\": rpc error: code = NotFound desc = could not find container \"2d701cb3cf9e41322d8b8d01265d13fdc81ef21fa27aa6d2f3423d7c0e6643df\": container with ID starting with 2d701cb3cf9e41322d8b8d01265d13fdc81ef21fa27aa6d2f3423d7c0e6643df not found: ID does not exist"
Nov 03 02:25:19.001784 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:19.001750    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"]
Nov 03 02:25:19.004884 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:19.004859    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-h4htz"]
Nov 03 02:25:19.109876 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:19.109798    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af839e9e-267b-4c27-b8cc-4e4e591580e3" path="/var/lib/kubelet/pods/af839e9e-267b-4c27-b8cc-4e4e591580e3/volumes"
Nov 03 02:25:25.660304 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:25.660266    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"]
Nov 03 02:25:25.660676 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:25.660480    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8" podUID="a205d365-889f-4706-9967-3e98bc1fc7d2" containerName="maas-api" containerID="cri-o://4dc46c05ba2c9aa3555a0554dafc4e7ce8d05a8346bd42cb042e72a7c27448a4" gracePeriod=30
Nov 03 02:25:25.908434 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:25.907232    5005 generic.go:356] "Generic (PLEG): container finished" podID="a205d365-889f-4706-9967-3e98bc1fc7d2" containerID="4dc46c05ba2c9aa3555a0554dafc4e7ce8d05a8346bd42cb042e72a7c27448a4" exitCode=0
Nov 03 02:25:25.908434 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:25.907363    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8" event={"ID":"a205d365-889f-4706-9967-3e98bc1fc7d2","Type":"ContainerDied","Data":"4dc46c05ba2c9aa3555a0554dafc4e7ce8d05a8346bd42cb042e72a7c27448a4"}
Nov 03 02:25:25.958365 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:25.958336    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"
Nov 03 02:25:25.993010 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:25.992982    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a205d365-889f-4706-9967-3e98bc1fc7d2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a205d365-889f-4706-9967-3e98bc1fc7d2-metrics-certs\") pod \"a205d365-889f-4706-9967-3e98bc1fc7d2\" (UID: \"a205d365-889f-4706-9967-3e98bc1fc7d2\") "
Nov 03 02:25:25.993174 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:25.993019    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a205d365-889f-4706-9967-3e98bc1fc7d2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a205d365-889f-4706-9967-3e98bc1fc7d2-openshift-service-ca\") pod \"a205d365-889f-4706-9967-3e98bc1fc7d2\" (UID: \"a205d365-889f-4706-9967-3e98bc1fc7d2\") "
Nov 03 02:25:25.993174 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:25.993061    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a205d365-889f-4706-9967-3e98bc1fc7d2-kube-api-access-dbwj9\" (UniqueName: \"kubernetes.io/projected/a205d365-889f-4706-9967-3e98bc1fc7d2-kube-api-access-dbwj9\") pod \"a205d365-889f-4706-9967-3e98bc1fc7d2\" (UID: \"a205d365-889f-4706-9967-3e98bc1fc7d2\") "
Nov 03 02:25:25.993174 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:25.993097    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a205d365-889f-4706-9967-3e98bc1fc7d2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a205d365-889f-4706-9967-3e98bc1fc7d2-maas-api-tls\") pod \"a205d365-889f-4706-9967-3e98bc1fc7d2\" (UID: \"a205d365-889f-4706-9967-3e98bc1fc7d2\") "
Nov 03 02:25:25.993467 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:25.993441    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a205d365-889f-4706-9967-3e98bc1fc7d2-openshift-service-ca" pod "a205d365-889f-4706-9967-3e98bc1fc7d2" (UID: "a205d365-889f-4706-9967-3e98bc1fc7d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:25:25.993779 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:25.993761    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a205d365-889f-4706-9967-3e98bc1fc7d2-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:25:25.995205 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:25.995176    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a205d365-889f-4706-9967-3e98bc1fc7d2-metrics-certs" pod "a205d365-889f-4706-9967-3e98bc1fc7d2" (UID: "a205d365-889f-4706-9967-3e98bc1fc7d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:25:25.995305 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:25.995202    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a205d365-889f-4706-9967-3e98bc1fc7d2-kube-api-access-dbwj9" pod "a205d365-889f-4706-9967-3e98bc1fc7d2" (UID: "a205d365-889f-4706-9967-3e98bc1fc7d2"). InnerVolumeSpecName "kube-api-access-dbwj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:25:25.995305 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:25.995210    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a205d365-889f-4706-9967-3e98bc1fc7d2-maas-api-tls" pod "a205d365-889f-4706-9967-3e98bc1fc7d2" (UID: "a205d365-889f-4706-9967-3e98bc1fc7d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:25:26.100715 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:26.100671    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a205d365-889f-4706-9967-3e98bc1fc7d2-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:25:26.100715 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:26.100719    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbwj9\" (UniqueName: \"kubernetes.io/projected/a205d365-889f-4706-9967-3e98bc1fc7d2-kube-api-access-dbwj9\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:25:26.100978 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:26.100732    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a205d365-889f-4706-9967-3e98bc1fc7d2-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:25:26.915252 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:26.915222    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"
Nov 03 02:25:26.915673 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:26.915223    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8" event={"ID":"a205d365-889f-4706-9967-3e98bc1fc7d2","Type":"ContainerDied","Data":"180391ed58ef5524e718a76364c8c86525abbf349741d5faf150111c70a2410b"}
Nov 03 02:25:26.915673 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:26.915353    5005 scope.go:122] "RemoveContainer" containerID="4dc46c05ba2c9aa3555a0554dafc4e7ce8d05a8346bd42cb042e72a7c27448a4"
Nov 03 02:25:27.069381 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:27.069346    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"]
Nov 03 02:25:27.072481 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:27.072457    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad4591-75b5db7578-8mpv8"]
Nov 03 02:25:27.110071 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:27.110040    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a205d365-889f-4706-9967-3e98bc1fc7d2" path="/var/lib/kubelet/pods/a205d365-889f-4706-9967-3e98bc1fc7d2/volumes"
Nov 03 02:25:30.715225 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:25:30.715187    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:25:42.573297 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:42.573261    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"]
Nov 03 02:25:42.576644 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:42.573509    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d" podUID="88c68e02-3246-440d-b299-ab6c10bf4258" containerName="maas-api" containerID="cri-o://120d801c49f54929c7928596296b7def6476bbf97cb98da6a218397d78344c58" gracePeriod=30
Nov 03 02:25:42.923755 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:42.923723    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"
Nov 03 02:25:43.026802 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:43.026771    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88c68e02-3246-440d-b299-ab6c10bf4258-kube-api-access-j4ckb\" (UniqueName: \"kubernetes.io/projected/88c68e02-3246-440d-b299-ab6c10bf4258-kube-api-access-j4ckb\") pod \"88c68e02-3246-440d-b299-ab6c10bf4258\" (UID: \"88c68e02-3246-440d-b299-ab6c10bf4258\") "
Nov 03 02:25:43.027013 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:43.026824    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88c68e02-3246-440d-b299-ab6c10bf4258-metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c68e02-3246-440d-b299-ab6c10bf4258-metrics-certs\") pod \"88c68e02-3246-440d-b299-ab6c10bf4258\" (UID: \"88c68e02-3246-440d-b299-ab6c10bf4258\") "
Nov 03 02:25:43.027013 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:43.026933    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88c68e02-3246-440d-b299-ab6c10bf4258-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88c68e02-3246-440d-b299-ab6c10bf4258-openshift-service-ca\") pod \"88c68e02-3246-440d-b299-ab6c10bf4258\" (UID: \"88c68e02-3246-440d-b299-ab6c10bf4258\") "
Nov 03 02:25:43.027139 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:43.027078    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88c68e02-3246-440d-b299-ab6c10bf4258-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88c68e02-3246-440d-b299-ab6c10bf4258-maas-api-tls\") pod \"88c68e02-3246-440d-b299-ab6c10bf4258\" (UID: \"88c68e02-3246-440d-b299-ab6c10bf4258\") "
Nov 03 02:25:43.027344 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:43.027302    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88c68e02-3246-440d-b299-ab6c10bf4258-openshift-service-ca" pod "88c68e02-3246-440d-b299-ab6c10bf4258" (UID: "88c68e02-3246-440d-b299-ab6c10bf4258"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:25:43.027597 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:43.027575    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88c68e02-3246-440d-b299-ab6c10bf4258-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:25:43.029424 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:43.029397    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88c68e02-3246-440d-b299-ab6c10bf4258-metrics-certs" pod "88c68e02-3246-440d-b299-ab6c10bf4258" (UID: "88c68e02-3246-440d-b299-ab6c10bf4258"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:25:43.029557 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:43.029535    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88c68e02-3246-440d-b299-ab6c10bf4258-maas-api-tls" pod "88c68e02-3246-440d-b299-ab6c10bf4258" (UID: "88c68e02-3246-440d-b299-ab6c10bf4258"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:25:43.029622 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:43.029540    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c68e02-3246-440d-b299-ab6c10bf4258-kube-api-access-j4ckb" pod "88c68e02-3246-440d-b299-ab6c10bf4258" (UID: "88c68e02-3246-440d-b299-ab6c10bf4258"). InnerVolumeSpecName "kube-api-access-j4ckb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:25:43.035854 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:43.033916    5005 generic.go:356] "Generic (PLEG): container finished" podID="88c68e02-3246-440d-b299-ab6c10bf4258" containerID="120d801c49f54929c7928596296b7def6476bbf97cb98da6a218397d78344c58" exitCode=0
Nov 03 02:25:43.035854 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:43.034072    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d" event={"ID":"88c68e02-3246-440d-b299-ab6c10bf4258","Type":"ContainerDied","Data":"120d801c49f54929c7928596296b7def6476bbf97cb98da6a218397d78344c58"}
Nov 03 02:25:43.035854 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:43.034123    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d" event={"ID":"88c68e02-3246-440d-b299-ab6c10bf4258","Type":"ContainerDied","Data":"e42d7fa4c19b33d244b74190f82719850eb009ccd3a467bb729708550e3ed410"}
Nov 03 02:25:43.035854 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:43.034133    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"
Nov 03 02:25:43.035854 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:43.034148    5005 scope.go:122] "RemoveContainer" containerID="120d801c49f54929c7928596296b7def6476bbf97cb98da6a218397d78344c58"
Nov 03 02:25:43.079783 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:43.070884    5005 scope.go:122] "RemoveContainer" containerID="120d801c49f54929c7928596296b7def6476bbf97cb98da6a218397d78344c58"
Nov 03 02:25:43.079783 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:25:43.071464    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"120d801c49f54929c7928596296b7def6476bbf97cb98da6a218397d78344c58\": container with ID starting with 120d801c49f54929c7928596296b7def6476bbf97cb98da6a218397d78344c58 not found: ID does not exist" containerID="120d801c49f54929c7928596296b7def6476bbf97cb98da6a218397d78344c58"
Nov 03 02:25:43.079783 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:43.071495    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"120d801c49f54929c7928596296b7def6476bbf97cb98da6a218397d78344c58"} err="failed to get container status \"120d801c49f54929c7928596296b7def6476bbf97cb98da6a218397d78344c58\": rpc error: code = NotFound desc = could not find container \"120d801c49f54929c7928596296b7def6476bbf97cb98da6a218397d78344c58\": container with ID starting with 120d801c49f54929c7928596296b7def6476bbf97cb98da6a218397d78344c58 not found: ID does not exist"
Nov 03 02:25:43.129219 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:43.129187    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c68e02-3246-440d-b299-ab6c10bf4258-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:25:43.129219 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:43.129213    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88c68e02-3246-440d-b299-ab6c10bf4258-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:25:43.129219 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:43.129225    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4ckb\" (UniqueName: \"kubernetes.io/projected/88c68e02-3246-440d-b299-ab6c10bf4258-kube-api-access-j4ckb\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:25:43.189973 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:43.189884    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"]
Nov 03 02:25:43.192959 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:43.192930    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3bbfc4-69ccf47898-fcj9d"]
Nov 03 02:25:45.109557 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:25:45.109519    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88c68e02-3246-440d-b299-ab6c10bf4258" path="/var/lib/kubelet/pods/88c68e02-3246-440d-b299-ab6c10bf4258/volumes"
Nov 03 02:26:00.714144 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:26:00.714104    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:26:10.903902 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:10.903867    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"]
Nov 03 02:26:10.904390 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:10.904093    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc" podUID="8e19ed57-84d2-499a-8a62-3b5beac110a2" containerName="istio-proxy" containerID="cri-o://826c755b33d9929f5ae76165cee379d8245ab5b20a824900a11c118fa56a698e" gracePeriod=30
Nov 03 02:26:13.399200 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:13.399170    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:26:13.409957 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:13.409934    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:26:14.786660 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:14.786619    5005 patch_prober.go:29] interesting pod/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 02:26:14.787285 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:14.786678    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc" podUID="8e19ed57-84d2-499a-8a62-3b5beac110a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 02:26:15.647852 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:15.647805    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"]
Nov 03 02:26:15.648119 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:15.648068    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl" podUID="30ce4650-88d3-4c9e-a333-7fdbea243a40" containerName="maas-api" containerID="cri-o://a8e801f5f0b62a02a7bdfb9cf3226c70f632ed7482b871029b37d51452022572" gracePeriod=30
Nov 03 02:26:16.057498 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.057466    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"
Nov 03 02:26:16.167292 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.167255    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30ce4650-88d3-4c9e-a333-7fdbea243a40-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ce4650-88d3-4c9e-a333-7fdbea243a40-maas-api-tls\") pod \"30ce4650-88d3-4c9e-a333-7fdbea243a40\" (UID: \"30ce4650-88d3-4c9e-a333-7fdbea243a40\") "
Nov 03 02:26:16.167491 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.167475    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30ce4650-88d3-4c9e-a333-7fdbea243a40-kube-api-access-tddlf\" (UniqueName: \"kubernetes.io/projected/30ce4650-88d3-4c9e-a333-7fdbea243a40-kube-api-access-tddlf\") pod \"30ce4650-88d3-4c9e-a333-7fdbea243a40\" (UID: \"30ce4650-88d3-4c9e-a333-7fdbea243a40\") "
Nov 03 02:26:16.167564 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.167548    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30ce4650-88d3-4c9e-a333-7fdbea243a40-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ce4650-88d3-4c9e-a333-7fdbea243a40-openshift-service-ca\") pod \"30ce4650-88d3-4c9e-a333-7fdbea243a40\" (UID: \"30ce4650-88d3-4c9e-a333-7fdbea243a40\") "
Nov 03 02:26:16.167636 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.167593    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30ce4650-88d3-4c9e-a333-7fdbea243a40-metrics-certs\" (UniqueName: \"kubernetes.io/secret/30ce4650-88d3-4c9e-a333-7fdbea243a40-metrics-certs\") pod \"30ce4650-88d3-4c9e-a333-7fdbea243a40\" (UID: \"30ce4650-88d3-4c9e-a333-7fdbea243a40\") "
Nov 03 02:26:16.167993 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.167931    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ce4650-88d3-4c9e-a333-7fdbea243a40-openshift-service-ca" pod "30ce4650-88d3-4c9e-a333-7fdbea243a40" (UID: "30ce4650-88d3-4c9e-a333-7fdbea243a40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:26:16.168170 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.168153    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ce4650-88d3-4c9e-a333-7fdbea243a40-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:16.169541 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.169519    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ce4650-88d3-4c9e-a333-7fdbea243a40-kube-api-access-tddlf" pod "30ce4650-88d3-4c9e-a333-7fdbea243a40" (UID: "30ce4650-88d3-4c9e-a333-7fdbea243a40"). InnerVolumeSpecName "kube-api-access-tddlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:26:16.169807 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.169776    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ce4650-88d3-4c9e-a333-7fdbea243a40-maas-api-tls" pod "30ce4650-88d3-4c9e-a333-7fdbea243a40" (UID: "30ce4650-88d3-4c9e-a333-7fdbea243a40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:26:16.169911 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.169846    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ce4650-88d3-4c9e-a333-7fdbea243a40-metrics-certs" pod "30ce4650-88d3-4c9e-a333-7fdbea243a40" (UID: "30ce4650-88d3-4c9e-a333-7fdbea243a40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:26:16.228166 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.211864    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:26:16.268531 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.268500    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-workload-certs\") pod \"8e19ed57-84d2-499a-8a62-3b5beac110a2\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") "
Nov 03 02:26:16.268759 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.268601    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-workload-socket\") pod \"8e19ed57-84d2-499a-8a62-3b5beac110a2\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") "
Nov 03 02:26:16.268759 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.268671    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-podinfo\") pod \"8e19ed57-84d2-499a-8a62-3b5beac110a2\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") "
Nov 03 02:26:16.268759 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.268733    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-credential-socket\") pod \"8e19ed57-84d2-499a-8a62-3b5beac110a2\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") "
Nov 03 02:26:16.268957 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.268817    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-token\" (UniqueName: \"kubernetes.io/projected/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-token\") pod \"8e19ed57-84d2-499a-8a62-3b5beac110a2\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") "
Nov 03 02:26:16.268957 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.268835    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-workload-certs" pod "8e19ed57-84d2-499a-8a62-3b5beac110a2" (UID: "8e19ed57-84d2-499a-8a62-3b5beac110a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:26:16.268957 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.268850    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-data\") pod \"8e19ed57-84d2-499a-8a62-3b5beac110a2\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") "
Nov 03 02:26:16.268957 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.268892    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-envoy\") pod \"8e19ed57-84d2-499a-8a62-3b5beac110a2\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") "
Nov 03 02:26:16.269299 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.268980    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e19ed57-84d2-499a-8a62-3b5beac110a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e19ed57-84d2-499a-8a62-3b5beac110a2-istiod-ca-cert\") pod \"8e19ed57-84d2-499a-8a62-3b5beac110a2\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") "
Nov 03 02:26:16.269299 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.269017    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e19ed57-84d2-499a-8a62-3b5beac110a2-kube-api-access-7rblt\" (UniqueName: \"kubernetes.io/projected/8e19ed57-84d2-499a-8a62-3b5beac110a2-kube-api-access-7rblt\") pod \"8e19ed57-84d2-499a-8a62-3b5beac110a2\" (UID: \"8e19ed57-84d2-499a-8a62-3b5beac110a2\") "
Nov 03 02:26:16.269299 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.269032    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-credential-socket" pod "8e19ed57-84d2-499a-8a62-3b5beac110a2" (UID: "8e19ed57-84d2-499a-8a62-3b5beac110a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:26:16.269299 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.269261    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-workload-socket" pod "8e19ed57-84d2-499a-8a62-3b5beac110a2" (UID: "8e19ed57-84d2-499a-8a62-3b5beac110a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:26:16.269660 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.269636    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tddlf\" (UniqueName: \"kubernetes.io/projected/30ce4650-88d3-4c9e-a333-7fdbea243a40-kube-api-access-tddlf\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:16.269660 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.269649    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e19ed57-84d2-499a-8a62-3b5beac110a2-istiod-ca-cert" pod "8e19ed57-84d2-499a-8a62-3b5beac110a2" (UID: "8e19ed57-84d2-499a-8a62-3b5beac110a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:26:16.269920 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.269667    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30ce4650-88d3-4c9e-a333-7fdbea243a40-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:16.269920 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.269681    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:16.269920 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.269711    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:16.269920 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.269727    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ce4650-88d3-4c9e-a333-7fdbea243a40-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:16.269920 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.269739    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:16.270170 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.270037    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-data" pod "8e19ed57-84d2-499a-8a62-3b5beac110a2" (UID: "8e19ed57-84d2-499a-8a62-3b5beac110a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:26:16.272107 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.272073    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-token" pod "8e19ed57-84d2-499a-8a62-3b5beac110a2" (UID: "8e19ed57-84d2-499a-8a62-3b5beac110a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:26:16.272230 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.272183    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-podinfo" pod "8e19ed57-84d2-499a-8a62-3b5beac110a2" (UID: "8e19ed57-84d2-499a-8a62-3b5beac110a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:26:16.272286 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.272229    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e19ed57-84d2-499a-8a62-3b5beac110a2-kube-api-access-7rblt" pod "8e19ed57-84d2-499a-8a62-3b5beac110a2" (UID: "8e19ed57-84d2-499a-8a62-3b5beac110a2"). InnerVolumeSpecName "kube-api-access-7rblt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:26:16.273122 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.273105    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-envoy" pod "8e19ed57-84d2-499a-8a62-3b5beac110a2" (UID: "8e19ed57-84d2-499a-8a62-3b5beac110a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:26:16.285055 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.285031    5005 generic.go:356] "Generic (PLEG): container finished" podID="8e19ed57-84d2-499a-8a62-3b5beac110a2" containerID="826c755b33d9929f5ae76165cee379d8245ab5b20a824900a11c118fa56a698e" exitCode=0
Nov 03 02:26:16.285154 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.285099    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"
Nov 03 02:26:16.285154 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.285117    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc" event={"ID":"8e19ed57-84d2-499a-8a62-3b5beac110a2","Type":"ContainerDied","Data":"826c755b33d9929f5ae76165cee379d8245ab5b20a824900a11c118fa56a698e"}
Nov 03 02:26:16.285265 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.285169    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc" event={"ID":"8e19ed57-84d2-499a-8a62-3b5beac110a2","Type":"ContainerDied","Data":"77ae09c334b847c8edb47a582034529333a79fde8e1b0496a96f44b20ddcd5dc"}
Nov 03 02:26:16.285265 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.285196    5005 scope.go:122] "RemoveContainer" containerID="826c755b33d9929f5ae76165cee379d8245ab5b20a824900a11c118fa56a698e"
Nov 03 02:26:16.286508 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.286484    5005 generic.go:356] "Generic (PLEG): container finished" podID="30ce4650-88d3-4c9e-a333-7fdbea243a40" containerID="a8e801f5f0b62a02a7bdfb9cf3226c70f632ed7482b871029b37d51452022572" exitCode=0
Nov 03 02:26:16.286599 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.286519    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"
Nov 03 02:26:16.286599 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.286556    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl" event={"ID":"30ce4650-88d3-4c9e-a333-7fdbea243a40","Type":"ContainerDied","Data":"a8e801f5f0b62a02a7bdfb9cf3226c70f632ed7482b871029b37d51452022572"}
Nov 03 02:26:16.286745 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.286596    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl" event={"ID":"30ce4650-88d3-4c9e-a333-7fdbea243a40","Type":"ContainerDied","Data":"b001ab33c6a86ef350226502eea67c7ff88ef880b27c625d68f7e06f0de2ca09"}
Nov 03 02:26:16.295935 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.295915    5005 scope.go:122] "RemoveContainer" containerID="826c755b33d9929f5ae76165cee379d8245ab5b20a824900a11c118fa56a698e"
Nov 03 02:26:16.299849 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:26:16.296933    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"826c755b33d9929f5ae76165cee379d8245ab5b20a824900a11c118fa56a698e\": container with ID starting with 826c755b33d9929f5ae76165cee379d8245ab5b20a824900a11c118fa56a698e not found: ID does not exist" containerID="826c755b33d9929f5ae76165cee379d8245ab5b20a824900a11c118fa56a698e"
Nov 03 02:26:16.299849 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.297057    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"826c755b33d9929f5ae76165cee379d8245ab5b20a824900a11c118fa56a698e"} err="failed to get container status \"826c755b33d9929f5ae76165cee379d8245ab5b20a824900a11c118fa56a698e\": rpc error: code = NotFound desc = could not find container \"826c755b33d9929f5ae76165cee379d8245ab5b20a824900a11c118fa56a698e\": container with ID starting with 826c755b33d9929f5ae76165cee379d8245ab5b20a824900a11c118fa56a698e not found: ID does not exist"
Nov 03 02:26:16.299849 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.297080    5005 scope.go:122] "RemoveContainer" containerID="a8e801f5f0b62a02a7bdfb9cf3226c70f632ed7482b871029b37d51452022572"
Nov 03 02:26:16.307287 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.307269    5005 scope.go:122] "RemoveContainer" containerID="a8e801f5f0b62a02a7bdfb9cf3226c70f632ed7482b871029b37d51452022572"
Nov 03 02:26:16.307518 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:26:16.307500    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e801f5f0b62a02a7bdfb9cf3226c70f632ed7482b871029b37d51452022572\": container with ID starting with a8e801f5f0b62a02a7bdfb9cf3226c70f632ed7482b871029b37d51452022572 not found: ID does not exist" containerID="a8e801f5f0b62a02a7bdfb9cf3226c70f632ed7482b871029b37d51452022572"
Nov 03 02:26:16.307568 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.307524    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e801f5f0b62a02a7bdfb9cf3226c70f632ed7482b871029b37d51452022572"} err="failed to get container status \"a8e801f5f0b62a02a7bdfb9cf3226c70f632ed7482b871029b37d51452022572\": rpc error: code = NotFound desc = could not find container \"a8e801f5f0b62a02a7bdfb9cf3226c70f632ed7482b871029b37d51452022572\": container with ID starting with a8e801f5f0b62a02a7bdfb9cf3226c70f632ed7482b871029b37d51452022572 not found: ID does not exist"
Nov 03 02:26:16.370759 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.370728    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:16.370759 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.370752    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:16.370759 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.370763    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:16.371008 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.370772    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e19ed57-84d2-499a-8a62-3b5beac110a2-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:16.371008 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.370782    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e19ed57-84d2-499a-8a62-3b5beac110a2-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:16.371008 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.370790    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rblt\" (UniqueName: \"kubernetes.io/projected/8e19ed57-84d2-499a-8a62-3b5beac110a2-kube-api-access-7rblt\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:16.426077 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.426001    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"]
Nov 03 02:26:16.428961 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.428937    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bd3ab-58f9d66d7f-5h7fl"]
Nov 03 02:26:16.529457 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.529415    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"]
Nov 03 02:26:16.532719 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:16.532676    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-785wc"]
Nov 03 02:26:17.109715 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:17.109674    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30ce4650-88d3-4c9e-a333-7fdbea243a40" path="/var/lib/kubelet/pods/30ce4650-88d3-4c9e-a333-7fdbea243a40/volumes"
Nov 03 02:26:17.110209 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:17.110194    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e19ed57-84d2-499a-8a62-3b5beac110a2" path="/var/lib/kubelet/pods/8e19ed57-84d2-499a-8a62-3b5beac110a2/volumes"
Nov 03 02:26:19.306201 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:19.306168    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"]
Nov 03 02:26:19.306732 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:19.306399    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277" podUID="d0ba8522-d62e-4f2e-a556-68599dd34e95" containerName="istio-proxy" containerID="cri-o://0e48605739173628a92ac95558f5c74152fb783f317144d8738b256f9e768202" gracePeriod=30
Nov 03 02:26:20.089854 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:20.089814    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 02:26:20.090039 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:20.089885    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277" podUID="d0ba8522-d62e-4f2e-a556-68599dd34e95" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 02:26:24.588852 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.588789    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:26:24.675879 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.675846    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0ba8522-d62e-4f2e-a556-68599dd34e95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0ba8522-d62e-4f2e-a556-68599dd34e95-istiod-ca-cert\") pod \"d0ba8522-d62e-4f2e-a556-68599dd34e95\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") "
Nov 03 02:26:24.676054 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.675941    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-workload-certs\") pod \"d0ba8522-d62e-4f2e-a556-68599dd34e95\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") "
Nov 03 02:26:24.676054 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.675972    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-data\") pod \"d0ba8522-d62e-4f2e-a556-68599dd34e95\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") "
Nov 03 02:26:24.676054 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.676030    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-credential-socket\") pod \"d0ba8522-d62e-4f2e-a556-68599dd34e95\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") "
Nov 03 02:26:24.676194 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.676065    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-workload-socket\") pod \"d0ba8522-d62e-4f2e-a556-68599dd34e95\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") "
Nov 03 02:26:24.676194 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.676120    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0ba8522-d62e-4f2e-a556-68599dd34e95-kube-api-access-2z9ks\" (UniqueName: \"kubernetes.io/projected/d0ba8522-d62e-4f2e-a556-68599dd34e95-kube-api-access-2z9ks\") pod \"d0ba8522-d62e-4f2e-a556-68599dd34e95\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") "
Nov 03 02:26:24.676307 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.676220    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-envoy\") pod \"d0ba8522-d62e-4f2e-a556-68599dd34e95\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") "
Nov 03 02:26:24.676307 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.676234    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-workload-certs" pod "d0ba8522-d62e-4f2e-a556-68599dd34e95" (UID: "d0ba8522-d62e-4f2e-a556-68599dd34e95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:26:24.676307 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.676259    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-podinfo\") pod \"d0ba8522-d62e-4f2e-a556-68599dd34e95\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") "
Nov 03 02:26:24.676307 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.676297    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-token\" (UniqueName: \"kubernetes.io/projected/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-token\") pod \"d0ba8522-d62e-4f2e-a556-68599dd34e95\" (UID: \"d0ba8522-d62e-4f2e-a556-68599dd34e95\") "
Nov 03 02:26:24.676500 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.676330    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-data" pod "d0ba8522-d62e-4f2e-a556-68599dd34e95" (UID: "d0ba8522-d62e-4f2e-a556-68599dd34e95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:26:24.676500 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.676358    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ba8522-d62e-4f2e-a556-68599dd34e95-istiod-ca-cert" pod "d0ba8522-d62e-4f2e-a556-68599dd34e95" (UID: "d0ba8522-d62e-4f2e-a556-68599dd34e95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:26:24.676500 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.676354    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-workload-socket" pod "d0ba8522-d62e-4f2e-a556-68599dd34e95" (UID: "d0ba8522-d62e-4f2e-a556-68599dd34e95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:26:24.676500 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.676398    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-credential-socket" pod "d0ba8522-d62e-4f2e-a556-68599dd34e95" (UID: "d0ba8522-d62e-4f2e-a556-68599dd34e95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:26:24.676801 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.676778    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0ba8522-d62e-4f2e-a556-68599dd34e95-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:24.676801 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.676803    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:24.677001 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.676816    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:24.677001 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.676830    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:24.677001 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.676841    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:24.678487 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.678465    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-envoy" pod "d0ba8522-d62e-4f2e-a556-68599dd34e95" (UID: "d0ba8522-d62e-4f2e-a556-68599dd34e95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:26:24.678623 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.678604    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-token" pod "d0ba8522-d62e-4f2e-a556-68599dd34e95" (UID: "d0ba8522-d62e-4f2e-a556-68599dd34e95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:26:24.678790 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.678772    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-podinfo" pod "d0ba8522-d62e-4f2e-a556-68599dd34e95" (UID: "d0ba8522-d62e-4f2e-a556-68599dd34e95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:26:24.678857 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.678807    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ba8522-d62e-4f2e-a556-68599dd34e95-kube-api-access-2z9ks" pod "d0ba8522-d62e-4f2e-a556-68599dd34e95" (UID: "d0ba8522-d62e-4f2e-a556-68599dd34e95"). InnerVolumeSpecName "kube-api-access-2z9ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:26:24.778295 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.778259    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z9ks\" (UniqueName: \"kubernetes.io/projected/d0ba8522-d62e-4f2e-a556-68599dd34e95-kube-api-access-2z9ks\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:24.778295 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.778289    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:24.778295 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.778299    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:24.778546 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:24.778307    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0ba8522-d62e-4f2e-a556-68599dd34e95-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:25.346518 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:25.346483    5005 generic.go:356] "Generic (PLEG): container finished" podID="d0ba8522-d62e-4f2e-a556-68599dd34e95" containerID="0e48605739173628a92ac95558f5c74152fb783f317144d8738b256f9e768202" exitCode=0
Nov 03 02:26:25.346710 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:25.346530    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"
Nov 03 02:26:25.346710 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:25.346565    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277" event={"ID":"d0ba8522-d62e-4f2e-a556-68599dd34e95","Type":"ContainerDied","Data":"0e48605739173628a92ac95558f5c74152fb783f317144d8738b256f9e768202"}
Nov 03 02:26:25.346710 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:25.346627    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277" event={"ID":"d0ba8522-d62e-4f2e-a556-68599dd34e95","Type":"ContainerDied","Data":"8fe5695245a94d022cba33402cc2e7a6bd36b1a4a7a69e4d1477f76bb067b37e"}
Nov 03 02:26:25.346710 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:25.346657    5005 scope.go:122] "RemoveContainer" containerID="0e48605739173628a92ac95558f5c74152fb783f317144d8738b256f9e768202"
Nov 03 02:26:25.356521 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:25.356502    5005 scope.go:122] "RemoveContainer" containerID="0e48605739173628a92ac95558f5c74152fb783f317144d8738b256f9e768202"
Nov 03 02:26:25.356787 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:26:25.356763    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e48605739173628a92ac95558f5c74152fb783f317144d8738b256f9e768202\": container with ID starting with 0e48605739173628a92ac95558f5c74152fb783f317144d8738b256f9e768202 not found: ID does not exist" containerID="0e48605739173628a92ac95558f5c74152fb783f317144d8738b256f9e768202"
Nov 03 02:26:25.356849 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:25.356799    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e48605739173628a92ac95558f5c74152fb783f317144d8738b256f9e768202"} err="failed to get container status \"0e48605739173628a92ac95558f5c74152fb783f317144d8738b256f9e768202\": rpc error: code = NotFound desc = could not find container \"0e48605739173628a92ac95558f5c74152fb783f317144d8738b256f9e768202\": container with ID starting with 0e48605739173628a92ac95558f5c74152fb783f317144d8738b256f9e768202 not found: ID does not exist"
Nov 03 02:26:25.547527 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:25.547494    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"]
Nov 03 02:26:25.593729 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:25.593680    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-lm277"]
Nov 03 02:26:27.115669 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:27.115637    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0ba8522-d62e-4f2e-a556-68599dd34e95" path="/var/lib/kubelet/pods/d0ba8522-d62e-4f2e-a556-68599dd34e95/volumes"
Nov 03 02:26:30.716379 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:26:30.716344    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:26:51.798937 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.798902    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"]
Nov 03 02:26:51.799454 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.799439    5005 cpu_manager.go:414] "containerMap: removing container" podUID="a205d365-889f-4706-9967-3e98bc1fc7d2" containerName="maas-api"
Nov 03 02:26:51.799502 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.799455    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a205d365-889f-4706-9967-3e98bc1fc7d2" containerName="maas-api"
Nov 03 02:26:51.799502 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.799469    5005 cpu_manager.go:414] "containerMap: removing container" podUID="30ce4650-88d3-4c9e-a333-7fdbea243a40" containerName="maas-api"
Nov 03 02:26:51.799502 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.799474    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30ce4650-88d3-4c9e-a333-7fdbea243a40" containerName="maas-api"
Nov 03 02:26:51.799502 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.799481    5005 cpu_manager.go:414] "containerMap: removing container" podUID="88c68e02-3246-440d-b299-ab6c10bf4258" containerName="maas-api"
Nov 03 02:26:51.799502 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.799486    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88c68e02-3246-440d-b299-ab6c10bf4258" containerName="maas-api"
Nov 03 02:26:51.799502 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.799500    5005 cpu_manager.go:414] "containerMap: removing container" podUID="8e19ed57-84d2-499a-8a62-3b5beac110a2" containerName="istio-proxy"
Nov 03 02:26:51.799727 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.799505    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e19ed57-84d2-499a-8a62-3b5beac110a2" containerName="istio-proxy"
Nov 03 02:26:51.799727 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.799517    5005 cpu_manager.go:414] "containerMap: removing container" podUID="af839e9e-267b-4c27-b8cc-4e4e591580e3" containerName="istio-proxy"
Nov 03 02:26:51.799727 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.799522    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af839e9e-267b-4c27-b8cc-4e4e591580e3" containerName="istio-proxy"
Nov 03 02:26:51.799727 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.799535    5005 cpu_manager.go:414] "containerMap: removing container" podUID="d0ba8522-d62e-4f2e-a556-68599dd34e95" containerName="istio-proxy"
Nov 03 02:26:51.799727 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.799540    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0ba8522-d62e-4f2e-a556-68599dd34e95" containerName="istio-proxy"
Nov 03 02:26:51.799727 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.799636    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="a205d365-889f-4706-9967-3e98bc1fc7d2" containerName="maas-api"
Nov 03 02:26:51.799727 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.799646    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="30ce4650-88d3-4c9e-a333-7fdbea243a40" containerName="maas-api"
Nov 03 02:26:51.799727 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.799652    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="88c68e02-3246-440d-b299-ab6c10bf4258" containerName="maas-api"
Nov 03 02:26:51.799727 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.799660    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e19ed57-84d2-499a-8a62-3b5beac110a2" containerName="istio-proxy"
Nov 03 02:26:51.799727 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.799666    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="af839e9e-267b-4c27-b8cc-4e4e591580e3" containerName="istio-proxy"
Nov 03 02:26:51.799727 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.799675    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0ba8522-d62e-4f2e-a556-68599dd34e95" containerName="istio-proxy"
Nov 03 02:26:51.803467 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.803443    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"
Nov 03 02:26:51.853242 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.853203    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-7f9d9d-serving-cert\""
Nov 03 02:26:51.902899 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.902869    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"]
Nov 03 02:26:51.907119 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.907092    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6762613-dfe0-4e64-b920-6c479e144d55-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66\" (UID: \"b6762613-dfe0-4e64-b920-6c479e144d55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"
Nov 03 02:26:51.907246 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.907157    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6762613-dfe0-4e64-b920-6c479e144d55-metrics-certs\") pod \"maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66\" (UID: \"b6762613-dfe0-4e64-b920-6c479e144d55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"
Nov 03 02:26:51.907289 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.907247    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6762613-dfe0-4e64-b920-6c479e144d55-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66\" (UID: \"b6762613-dfe0-4e64-b920-6c479e144d55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"
Nov 03 02:26:51.907338 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:51.907302    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqxgl\" (UniqueName: \"kubernetes.io/projected/b6762613-dfe0-4e64-b920-6c479e144d55-kube-api-access-jqxgl\") pod \"maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66\" (UID: \"b6762613-dfe0-4e64-b920-6c479e144d55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"
Nov 03 02:26:52.008362 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:52.008327    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6762613-dfe0-4e64-b920-6c479e144d55-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66\" (UID: \"b6762613-dfe0-4e64-b920-6c479e144d55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"
Nov 03 02:26:52.008568 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:52.008386    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6762613-dfe0-4e64-b920-6c479e144d55-metrics-certs\") pod \"maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66\" (UID: \"b6762613-dfe0-4e64-b920-6c479e144d55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"
Nov 03 02:26:52.008568 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:52.008419    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6762613-dfe0-4e64-b920-6c479e144d55-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66\" (UID: \"b6762613-dfe0-4e64-b920-6c479e144d55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"
Nov 03 02:26:52.008568 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:52.008458    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqxgl\" (UniqueName: \"kubernetes.io/projected/b6762613-dfe0-4e64-b920-6c479e144d55-kube-api-access-jqxgl\") pod \"maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66\" (UID: \"b6762613-dfe0-4e64-b920-6c479e144d55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"
Nov 03 02:26:52.008568 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:26:52.008504    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-serving-cert: secret "maas-api-e2e-praxis-b-7f9d9d-serving-cert" not found
Nov 03 02:26:52.008853 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:26:52.008589    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6762613-dfe0-4e64-b920-6c479e144d55-maas-api-tls podName:b6762613-dfe0-4e64-b920-6c479e144d55 nodeName:}" failed. No retries permitted until 2026-11-03 02:26:52.508566523 +0000 UTC m=+2139.522807886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b6762613-dfe0-4e64-b920-6c479e144d55-maas-api-tls") pod "maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66" (UID: "b6762613-dfe0-4e64-b920-6c479e144d55") : secret "maas-api-e2e-praxis-b-7f9d9d-serving-cert" not found
Nov 03 02:26:52.009273 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:52.009250    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6762613-dfe0-4e64-b920-6c479e144d55-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66\" (UID: \"b6762613-dfe0-4e64-b920-6c479e144d55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"
Nov 03 02:26:52.011135 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:52.011106    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6762613-dfe0-4e64-b920-6c479e144d55-metrics-certs\") pod \"maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66\" (UID: \"b6762613-dfe0-4e64-b920-6c479e144d55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"
Nov 03 02:26:52.062927 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:52.062799    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqxgl\" (UniqueName: \"kubernetes.io/projected/b6762613-dfe0-4e64-b920-6c479e144d55-kube-api-access-jqxgl\") pod \"maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66\" (UID: \"b6762613-dfe0-4e64-b920-6c479e144d55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"
Nov 03 02:26:52.513863 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:52.513827    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6762613-dfe0-4e64-b920-6c479e144d55-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66\" (UID: \"b6762613-dfe0-4e64-b920-6c479e144d55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"
Nov 03 02:26:52.516278 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:52.516253    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6762613-dfe0-4e64-b920-6c479e144d55-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66\" (UID: \"b6762613-dfe0-4e64-b920-6c479e144d55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"
Nov 03 02:26:52.716627 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:52.716595    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"
Nov 03 02:26:53.094019 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:53.093937    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"]
Nov 03 02:26:53.100523 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:26:53.100496    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6762613_dfe0_4e64_b920_6c479e144d55.slice/crio-5da4f434d412e7672c2bfa77dec8ed436d7890c567ef8f682c46bb7dba71f335 WatchSource:0}: Error finding container 5da4f434d412e7672c2bfa77dec8ed436d7890c567ef8f682c46bb7dba71f335: Status 404 returned error can't find the container with id 5da4f434d412e7672c2bfa77dec8ed436d7890c567ef8f682c46bb7dba71f335
Nov 03 02:26:53.518348 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:53.518311    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66" event={"ID":"b6762613-dfe0-4e64-b920-6c479e144d55","Type":"ContainerStarted","Data":"5da4f434d412e7672c2bfa77dec8ed436d7890c567ef8f682c46bb7dba71f335"}
Nov 03 02:26:54.527411 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:54.527370    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66" event={"ID":"b6762613-dfe0-4e64-b920-6c479e144d55","Type":"ContainerStarted","Data":"c2a8d3e4c498bc942aa45ad33c22fff919078c39d08f2dbc3706e246c1820eee"}
Nov 03 02:26:54.527865 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:54.527518    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"
Nov 03 02:26:54.636245 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:26:54.636190    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66" podStartSLOduration=2.818709616 podStartE2EDuration="3.636176309s" totalImagesPullingTime="817.466693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:26:54.6339476 +0000 UTC m=+2141.648188983" watchObservedRunningTime="2026-11-03 02:26:54.636176309 +0000 UTC m=+2141.650417721"
Nov 03 02:27:00.539128 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:00.539093    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"
Nov 03 02:27:00.731404 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:27:00.731359    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:27:30.713513 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:27:30.713474    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:27:35.641172 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:35.641136    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"]
Nov 03 02:27:35.646825 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:35.646804    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"
Nov 03 02:27:35.700226 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:35.700195    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"]
Nov 03 02:27:35.742050 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:35.742021    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-7f760c-serving-cert\""
Nov 03 02:27:35.744607 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:35.744577    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv\" (UID: \"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"
Nov 03 02:27:35.744787 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:35.744757    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5rhj\" (UniqueName: \"kubernetes.io/projected/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-kube-api-access-f5rhj\") pod \"maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv\" (UID: \"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"
Nov 03 02:27:35.744871 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:35.744824    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv\" (UID: \"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"
Nov 03 02:27:35.744932 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:35.744918    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-metrics-certs\") pod \"maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv\" (UID: \"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"
Nov 03 02:27:35.846560 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:35.846523    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5rhj\" (UniqueName: \"kubernetes.io/projected/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-kube-api-access-f5rhj\") pod \"maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv\" (UID: \"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"
Nov 03 02:27:35.846782 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:35.846575    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv\" (UID: \"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"
Nov 03 02:27:35.846782 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:35.846621    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-metrics-certs\") pod \"maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv\" (UID: \"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"
Nov 03 02:27:35.846782 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:35.846680    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv\" (UID: \"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"
Nov 03 02:27:35.846991 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:27:35.846881    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-serving-cert: secret "maas-api-e2e-praxis-a-7f760c-serving-cert" not found
Nov 03 02:27:35.846991 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:27:35.846956    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-maas-api-tls podName:10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f nodeName:}" failed. No retries permitted until 2026-11-03 02:27:36.346934993 +0000 UTC m=+2183.361176337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-maas-api-tls") pod "maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv" (UID: "10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f") : secret "maas-api-e2e-praxis-a-7f760c-serving-cert" not found
Nov 03 02:27:35.847392 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:35.847369    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv\" (UID: \"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"
Nov 03 02:27:35.849316 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:35.849296    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-metrics-certs\") pod \"maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv\" (UID: \"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"
Nov 03 02:27:35.907102 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:35.903883    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5rhj\" (UniqueName: \"kubernetes.io/projected/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-kube-api-access-f5rhj\") pod \"maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv\" (UID: \"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"
Nov 03 02:27:36.352361 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:36.352325    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv\" (UID: \"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"
Nov 03 02:27:36.354831 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:36.354808    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv\" (UID: \"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"
Nov 03 02:27:36.559834 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:36.559797    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"
Nov 03 02:27:36.937426 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:36.937350    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"]
Nov 03 02:27:36.942664 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:27:36.942629    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10cbe959_14ee_4e1c_b5ae_5e3bf92beb2f.slice/crio-89c06977b3d86dc700dc46710071493415fe691a132fae4ee279534313158012 WatchSource:0}: Error finding container 89c06977b3d86dc700dc46710071493415fe691a132fae4ee279534313158012: Status 404 returned error can't find the container with id 89c06977b3d86dc700dc46710071493415fe691a132fae4ee279534313158012
Nov 03 02:27:37.802466 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:37.802412    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv" event={"ID":"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f","Type":"ContainerStarted","Data":"89c06977b3d86dc700dc46710071493415fe691a132fae4ee279534313158012"}
Nov 03 02:27:38.811058 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:38.811016    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv" event={"ID":"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f","Type":"ContainerStarted","Data":"09cddefe9d374538ec8956d820e87418498fc1c0f6844498b9571c43102029a7"}
Nov 03 02:27:38.811490 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:38.811167    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"
Nov 03 02:27:38.917897 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:38.917833    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv" podStartSLOduration=3.087577522 podStartE2EDuration="3.917819646s" totalImagesPullingTime="830.242124ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:27:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:27:38.917330411 +0000 UTC m=+2185.931571777" watchObservedRunningTime="2026-11-03 02:27:38.917819646 +0000 UTC m=+2185.932061012"
Nov 03 02:27:44.824534 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:27:44.824505    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"
Nov 03 02:28:00.714842 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:28:00.714759    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:28:30.714803 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:28:30.714767    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:28:37.261466 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:37.261430    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"]
Nov 03 02:28:37.262044 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:37.261638    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6" podUID="ffe1671a-c31c-4dc0-85a2-51d0b97daa81" containerName="maas-api" containerID="cri-o://ef140c4025083d0c33e88cf8279d1bbd3106363ee44a7425e96aaf275756bff5" gracePeriod=30
Nov 03 02:28:37.554164 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:37.554139    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"
Nov 03 02:28:37.633070 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:37.633031    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-metrics-certs\") pod \"ffe1671a-c31c-4dc0-85a2-51d0b97daa81\" (UID: \"ffe1671a-c31c-4dc0-85a2-51d0b97daa81\") "
Nov 03 02:28:37.633236 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:37.633086    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-openshift-service-ca\") pod \"ffe1671a-c31c-4dc0-85a2-51d0b97daa81\" (UID: \"ffe1671a-c31c-4dc0-85a2-51d0b97daa81\") "
Nov 03 02:28:37.633236 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:37.633150    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-maas-api-tls\") pod \"ffe1671a-c31c-4dc0-85a2-51d0b97daa81\" (UID: \"ffe1671a-c31c-4dc0-85a2-51d0b97daa81\") "
Nov 03 02:28:37.633322 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:37.633247    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-kube-api-access-qf8jd\" (UniqueName: \"kubernetes.io/projected/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-kube-api-access-qf8jd\") pod \"ffe1671a-c31c-4dc0-85a2-51d0b97daa81\" (UID: \"ffe1671a-c31c-4dc0-85a2-51d0b97daa81\") "
Nov 03 02:28:37.633487 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:37.633461    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-openshift-service-ca" pod "ffe1671a-c31c-4dc0-85a2-51d0b97daa81" (UID: "ffe1671a-c31c-4dc0-85a2-51d0b97daa81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:28:37.633729 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:37.633678    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:37.635378 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:37.635355    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-kube-api-access-qf8jd" pod "ffe1671a-c31c-4dc0-85a2-51d0b97daa81" (UID: "ffe1671a-c31c-4dc0-85a2-51d0b97daa81"). InnerVolumeSpecName "kube-api-access-qf8jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:28:37.635489 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:37.635355    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-metrics-certs" pod "ffe1671a-c31c-4dc0-85a2-51d0b97daa81" (UID: "ffe1671a-c31c-4dc0-85a2-51d0b97daa81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:28:37.635489 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:37.635383    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-maas-api-tls" pod "ffe1671a-c31c-4dc0-85a2-51d0b97daa81" (UID: "ffe1671a-c31c-4dc0-85a2-51d0b97daa81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:28:37.735988 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:37.735957    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:37.735988 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:37.735986    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf8jd\" (UniqueName: \"kubernetes.io/projected/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-kube-api-access-qf8jd\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:37.736191 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:37.735998    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffe1671a-c31c-4dc0-85a2-51d0b97daa81-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:38.226072 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:38.226036    5005 generic.go:356] "Generic (PLEG): container finished" podID="ffe1671a-c31c-4dc0-85a2-51d0b97daa81" containerID="ef140c4025083d0c33e88cf8279d1bbd3106363ee44a7425e96aaf275756bff5" exitCode=0
Nov 03 02:28:38.226389 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:38.226369    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6" event={"ID":"ffe1671a-c31c-4dc0-85a2-51d0b97daa81","Type":"ContainerDied","Data":"ef140c4025083d0c33e88cf8279d1bbd3106363ee44a7425e96aaf275756bff5"}
Nov 03 02:28:38.226531 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:38.226517    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6" event={"ID":"ffe1671a-c31c-4dc0-85a2-51d0b97daa81","Type":"ContainerDied","Data":"69f2149878b5d5b822e0a033a7a0df4043dd60a1e02d1cf8433fdc091f9954f5"}
Nov 03 02:28:38.226641 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:38.226629    5005 scope.go:122] "RemoveContainer" containerID="ef140c4025083d0c33e88cf8279d1bbd3106363ee44a7425e96aaf275756bff5"
Nov 03 02:28:38.226931 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:38.226916    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"
Nov 03 02:28:38.247431 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:38.247390    5005 scope.go:122] "RemoveContainer" containerID="ef140c4025083d0c33e88cf8279d1bbd3106363ee44a7425e96aaf275756bff5"
Nov 03 02:28:38.248162 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:28:38.248129    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef140c4025083d0c33e88cf8279d1bbd3106363ee44a7425e96aaf275756bff5\": container with ID starting with ef140c4025083d0c33e88cf8279d1bbd3106363ee44a7425e96aaf275756bff5 not found: ID does not exist" containerID="ef140c4025083d0c33e88cf8279d1bbd3106363ee44a7425e96aaf275756bff5"
Nov 03 02:28:38.248274 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:38.248175    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef140c4025083d0c33e88cf8279d1bbd3106363ee44a7425e96aaf275756bff5"} err="failed to get container status \"ef140c4025083d0c33e88cf8279d1bbd3106363ee44a7425e96aaf275756bff5\": rpc error: code = NotFound desc = could not find container \"ef140c4025083d0c33e88cf8279d1bbd3106363ee44a7425e96aaf275756bff5\": container with ID starting with ef140c4025083d0c33e88cf8279d1bbd3106363ee44a7425e96aaf275756bff5 not found: ID does not exist"
Nov 03 02:28:38.384180 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:38.384146    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"]
Nov 03 02:28:38.430766 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:38.430729    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6"]
Nov 03 02:28:39.110623 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:39.110591    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffe1671a-c31c-4dc0-85a2-51d0b97daa81" path="/var/lib/kubelet/pods/ffe1671a-c31c-4dc0-85a2-51d0b97daa81/volumes"
Nov 03 02:28:40.034002 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:40.033965    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"]
Nov 03 02:28:40.034515 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:40.034184    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z" podUID="d4394660-3dbc-46ea-9dd4-e4d52123648d" containerName="istio-proxy" containerID="cri-o://c8736a3da247a0131f454e8f034a940fe9ab54fd38b089e48256bda3d3b36a5d" gracePeriod=30
Nov 03 02:28:40.446311 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:40.446216    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c62c3d-56bd764668-xx5b6" podUID="ffe1671a-c31c-4dc0-85a2-51d0b97daa81" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.110:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:28:40.942023 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:40.941984    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 02:28:40.942188 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:40.942054    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z" podUID="d4394660-3dbc-46ea-9dd4-e4d52123648d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 02:28:45.270826 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.270792    5005 generic.go:356] "Generic (PLEG): container finished" podID="d4394660-3dbc-46ea-9dd4-e4d52123648d" containerID="c8736a3da247a0131f454e8f034a940fe9ab54fd38b089e48256bda3d3b36a5d" exitCode=0
Nov 03 02:28:45.271196 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.270862    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z" event={"ID":"d4394660-3dbc-46ea-9dd4-e4d52123648d","Type":"ContainerDied","Data":"c8736a3da247a0131f454e8f034a940fe9ab54fd38b089e48256bda3d3b36a5d"}
Nov 03 02:28:45.271196 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.270912    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z" event={"ID":"d4394660-3dbc-46ea-9dd4-e4d52123648d","Type":"ContainerDied","Data":"c877cd5a7d13cca90bfd64fa7990f964e7c9338a46140c18c2f1e99ba7875b46"}
Nov 03 02:28:45.271196 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.270928    5005 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c877cd5a7d13cca90bfd64fa7990f964e7c9338a46140c18c2f1e99ba7875b46"
Nov 03 02:28:45.290863 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.290840    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:28:45.347331 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.347265    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-token\" (UniqueName: \"kubernetes.io/projected/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-token\") pod \"d4394660-3dbc-46ea-9dd4-e4d52123648d\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") "
Nov 03 02:28:45.347331 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.347314    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-credential-socket\") pod \"d4394660-3dbc-46ea-9dd4-e4d52123648d\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") "
Nov 03 02:28:45.347533 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.347346    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-podinfo\") pod \"d4394660-3dbc-46ea-9dd4-e4d52123648d\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") "
Nov 03 02:28:45.347533 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.347385    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-workload-certs\") pod \"d4394660-3dbc-46ea-9dd4-e4d52123648d\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") "
Nov 03 02:28:45.347533 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.347404    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-envoy\") pod \"d4394660-3dbc-46ea-9dd4-e4d52123648d\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") "
Nov 03 02:28:45.347533 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.347522    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4394660-3dbc-46ea-9dd4-e4d52123648d-kube-api-access-wvhqp\" (UniqueName: \"kubernetes.io/projected/d4394660-3dbc-46ea-9dd4-e4d52123648d-kube-api-access-wvhqp\") pod \"d4394660-3dbc-46ea-9dd4-e4d52123648d\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") "
Nov 03 02:28:45.347778 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.347554    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-workload-socket\") pod \"d4394660-3dbc-46ea-9dd4-e4d52123648d\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") "
Nov 03 02:28:45.347778 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.347602    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4394660-3dbc-46ea-9dd4-e4d52123648d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4394660-3dbc-46ea-9dd4-e4d52123648d-istiod-ca-cert\") pod \"d4394660-3dbc-46ea-9dd4-e4d52123648d\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") "
Nov 03 02:28:45.347778 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.347668    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-data\") pod \"d4394660-3dbc-46ea-9dd4-e4d52123648d\" (UID: \"d4394660-3dbc-46ea-9dd4-e4d52123648d\") "
Nov 03 02:28:45.347778 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.347717    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-credential-socket" pod "d4394660-3dbc-46ea-9dd4-e4d52123648d" (UID: "d4394660-3dbc-46ea-9dd4-e4d52123648d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:28:45.347778 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.347685    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-workload-certs" pod "d4394660-3dbc-46ea-9dd4-e4d52123648d" (UID: "d4394660-3dbc-46ea-9dd4-e4d52123648d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:28:45.348042 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.347942    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-workload-socket" pod "d4394660-3dbc-46ea-9dd4-e4d52123648d" (UID: "d4394660-3dbc-46ea-9dd4-e4d52123648d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:28:45.348127 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.348057    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4394660-3dbc-46ea-9dd4-e4d52123648d-istiod-ca-cert" pod "d4394660-3dbc-46ea-9dd4-e4d52123648d" (UID: "d4394660-3dbc-46ea-9dd4-e4d52123648d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:28:45.348247 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.348153    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-data" pod "d4394660-3dbc-46ea-9dd4-e4d52123648d" (UID: "d4394660-3dbc-46ea-9dd4-e4d52123648d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:28:45.348247 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.348182    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:45.348247 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.348203    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:45.348247 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.348218    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:45.349674 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.349650    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-envoy" pod "d4394660-3dbc-46ea-9dd4-e4d52123648d" (UID: "d4394660-3dbc-46ea-9dd4-e4d52123648d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:28:45.349803 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.349672    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-podinfo" pod "d4394660-3dbc-46ea-9dd4-e4d52123648d" (UID: "d4394660-3dbc-46ea-9dd4-e4d52123648d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:28:45.349803 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.349672    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-token" pod "d4394660-3dbc-46ea-9dd4-e4d52123648d" (UID: "d4394660-3dbc-46ea-9dd4-e4d52123648d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:28:45.349960 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.349942    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4394660-3dbc-46ea-9dd4-e4d52123648d-kube-api-access-wvhqp" pod "d4394660-3dbc-46ea-9dd4-e4d52123648d" (UID: "d4394660-3dbc-46ea-9dd4-e4d52123648d"). InnerVolumeSpecName "kube-api-access-wvhqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:28:45.449326 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.449284    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:45.449326 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.449321    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:45.449580 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.449337    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:45.449580 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.449352    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4394660-3dbc-46ea-9dd4-e4d52123648d-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:45.449580 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.449366    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvhqp\" (UniqueName: \"kubernetes.io/projected/d4394660-3dbc-46ea-9dd4-e4d52123648d-kube-api-access-wvhqp\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:45.449580 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:45.449381    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4394660-3dbc-46ea-9dd4-e4d52123648d-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:46.277453 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:46.277418    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"
Nov 03 02:28:46.479097 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:46.479065    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"]
Nov 03 02:28:46.525442 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:46.525407    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-mnr9z"]
Nov 03 02:28:47.109987 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:47.109950    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4394660-3dbc-46ea-9dd4-e4d52123648d" path="/var/lib/kubelet/pods/d4394660-3dbc-46ea-9dd4-e4d52123648d/volumes"
Nov 03 02:28:59.015381 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.015345    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"]
Nov 03 02:28:59.016305 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.016273    5005 cpu_manager.go:414] "containerMap: removing container" podUID="d4394660-3dbc-46ea-9dd4-e4d52123648d" containerName="istio-proxy"
Nov 03 02:28:59.016305 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.016304    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4394660-3dbc-46ea-9dd4-e4d52123648d" containerName="istio-proxy"
Nov 03 02:28:59.016522 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.016324    5005 cpu_manager.go:414] "containerMap: removing container" podUID="ffe1671a-c31c-4dc0-85a2-51d0b97daa81" containerName="maas-api"
Nov 03 02:28:59.016522 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.016332    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe1671a-c31c-4dc0-85a2-51d0b97daa81" containerName="maas-api"
Nov 03 02:28:59.016522 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.016479    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4394660-3dbc-46ea-9dd4-e4d52123648d" containerName="istio-proxy"
Nov 03 02:28:59.016522 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.016492    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe1671a-c31c-4dc0-85a2-51d0b97daa81" containerName="maas-api"
Nov 03 02:28:59.021859 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.021834    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"
Nov 03 02:28:59.076312 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.076272    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"]
Nov 03 02:28:59.117771 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.117744    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-496177-serving-cert\""
Nov 03 02:28:59.135394 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.135360    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5npf2\" (UniqueName: \"kubernetes.io/projected/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-kube-api-access-5npf2\") pod \"maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh\" (UID: \"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"
Nov 03 02:28:59.135551 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.135530    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-metrics-certs\") pod \"maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh\" (UID: \"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"
Nov 03 02:28:59.135628 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.135574    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh\" (UID: \"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"
Nov 03 02:28:59.135687 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.135667    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh\" (UID: \"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"
Nov 03 02:28:59.236964 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.236924    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5npf2\" (UniqueName: \"kubernetes.io/projected/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-kube-api-access-5npf2\") pod \"maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh\" (UID: \"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"
Nov 03 02:28:59.237130 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.237055    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-metrics-certs\") pod \"maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh\" (UID: \"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"
Nov 03 02:28:59.237130 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.237078    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh\" (UID: \"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"
Nov 03 02:28:59.237130 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.237113    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh\" (UID: \"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"
Nov 03 02:28:59.237910 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.237882    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh\" (UID: \"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"
Nov 03 02:28:59.239578 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.239552    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-metrics-certs\") pod \"maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh\" (UID: \"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"
Nov 03 02:28:59.239733 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.239650    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh\" (UID: \"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"
Nov 03 02:28:59.291003 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.290919    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5npf2\" (UniqueName: \"kubernetes.io/projected/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-kube-api-access-5npf2\") pod \"maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh\" (UID: \"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"
Nov 03 02:28:59.336000 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.335968    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"
Nov 03 02:28:59.755452 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:28:59.755416    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"]
Nov 03 02:28:59.761628 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:28:59.761602    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda601098f_bdb7_402c_a5d2_4cfe04ff1ca4.slice/crio-e242a2781ab2694505aeb362d6b272eaba9e0e0391616412468ec93cb16b08e4 WatchSource:0}: Error finding container e242a2781ab2694505aeb362d6b272eaba9e0e0391616412468ec93cb16b08e4: Status 404 returned error can't find the container with id e242a2781ab2694505aeb362d6b272eaba9e0e0391616412468ec93cb16b08e4
Nov 03 02:29:00.454886 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:00.454842    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh" event={"ID":"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4","Type":"ContainerStarted","Data":"e242a2781ab2694505aeb362d6b272eaba9e0e0391616412468ec93cb16b08e4"}
Nov 03 02:29:00.705662 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:00.705637    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-dn4d4_30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2/payload-processing/0.log"
Nov 03 02:29:00.716911 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:29:00.716880    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:29:00.957107 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:00.957037    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-dn4d4_30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2/payload-processing/0.log"
Nov 03 02:29:01.462349 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:01.462307    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh" event={"ID":"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4","Type":"ContainerStarted","Data":"21e0cd42a67a111a0362d06eaaab957efe529f8e5bd965edb6a35692c2a0106a"}
Nov 03 02:29:01.462861 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:01.462530    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"
Nov 03 02:29:01.572079 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:01.572022    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh" podStartSLOduration=2.631383141 podStartE2EDuration="3.572005641s" totalImagesPullingTime="940.6225ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:29:01.570172998 +0000 UTC m=+2268.584414366" watchObservedRunningTime="2026-11-03 02:29:01.572005641 +0000 UTC m=+2268.586247008"
Nov 03 02:29:03.643576 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:03.643544    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"]
Nov 03 02:29:03.644054 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:03.643764    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66" podUID="b6762613-dfe0-4e64-b920-6c479e144d55" containerName="maas-api" containerID="cri-o://c2a8d3e4c498bc942aa45ad33c22fff919078c39d08f2dbc3706e246c1820eee" gracePeriod=30
Nov 03 02:29:03.933940 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:03.933910    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"
Nov 03 02:29:04.096337 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:04.096300    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6762613-dfe0-4e64-b920-6c479e144d55-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6762613-dfe0-4e64-b920-6c479e144d55-metrics-certs\") pod \"b6762613-dfe0-4e64-b920-6c479e144d55\" (UID: \"b6762613-dfe0-4e64-b920-6c479e144d55\") "
Nov 03 02:29:04.096527 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:04.096490    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6762613-dfe0-4e64-b920-6c479e144d55-kube-api-access-jqxgl\" (UniqueName: \"kubernetes.io/projected/b6762613-dfe0-4e64-b920-6c479e144d55-kube-api-access-jqxgl\") pod \"b6762613-dfe0-4e64-b920-6c479e144d55\" (UID: \"b6762613-dfe0-4e64-b920-6c479e144d55\") "
Nov 03 02:29:04.096615 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:04.096601    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6762613-dfe0-4e64-b920-6c479e144d55-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6762613-dfe0-4e64-b920-6c479e144d55-maas-api-tls\") pod \"b6762613-dfe0-4e64-b920-6c479e144d55\" (UID: \"b6762613-dfe0-4e64-b920-6c479e144d55\") "
Nov 03 02:29:04.096671 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:04.096654    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6762613-dfe0-4e64-b920-6c479e144d55-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6762613-dfe0-4e64-b920-6c479e144d55-openshift-service-ca\") pod \"b6762613-dfe0-4e64-b920-6c479e144d55\" (UID: \"b6762613-dfe0-4e64-b920-6c479e144d55\") "
Nov 03 02:29:04.097126 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:04.097081    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6762613-dfe0-4e64-b920-6c479e144d55-openshift-service-ca" pod "b6762613-dfe0-4e64-b920-6c479e144d55" (UID: "b6762613-dfe0-4e64-b920-6c479e144d55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:29:04.098517 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:04.098496    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6762613-dfe0-4e64-b920-6c479e144d55-metrics-certs" pod "b6762613-dfe0-4e64-b920-6c479e144d55" (UID: "b6762613-dfe0-4e64-b920-6c479e144d55"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:29:04.098747 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:04.098716    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6762613-dfe0-4e64-b920-6c479e144d55-maas-api-tls" pod "b6762613-dfe0-4e64-b920-6c479e144d55" (UID: "b6762613-dfe0-4e64-b920-6c479e144d55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:29:04.099040 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:04.099018    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6762613-dfe0-4e64-b920-6c479e144d55-kube-api-access-jqxgl" pod "b6762613-dfe0-4e64-b920-6c479e144d55" (UID: "b6762613-dfe0-4e64-b920-6c479e144d55"). InnerVolumeSpecName "kube-api-access-jqxgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:29:04.198916 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:04.198837    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqxgl\" (UniqueName: \"kubernetes.io/projected/b6762613-dfe0-4e64-b920-6c479e144d55-kube-api-access-jqxgl\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:29:04.198916 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:04.198867    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6762613-dfe0-4e64-b920-6c479e144d55-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:29:04.198916 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:04.198878    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6762613-dfe0-4e64-b920-6c479e144d55-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:29:04.198916 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:04.198887    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6762613-dfe0-4e64-b920-6c479e144d55-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:29:04.483809 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:04.483725    5005 generic.go:356] "Generic (PLEG): container finished" podID="b6762613-dfe0-4e64-b920-6c479e144d55" containerID="c2a8d3e4c498bc942aa45ad33c22fff919078c39d08f2dbc3706e246c1820eee" exitCode=0
Nov 03 02:29:04.483809 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:04.483767    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66" event={"ID":"b6762613-dfe0-4e64-b920-6c479e144d55","Type":"ContainerDied","Data":"c2a8d3e4c498bc942aa45ad33c22fff919078c39d08f2dbc3706e246c1820eee"}
Nov 03 02:29:04.484058 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:04.483837    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66" event={"ID":"b6762613-dfe0-4e64-b920-6c479e144d55","Type":"ContainerDied","Data":"5da4f434d412e7672c2bfa77dec8ed436d7890c567ef8f682c46bb7dba71f335"}
Nov 03 02:29:04.484058 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:04.483783    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"
Nov 03 02:29:04.484058 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:04.483869    5005 scope.go:122] "RemoveContainer" containerID="c2a8d3e4c498bc942aa45ad33c22fff919078c39d08f2dbc3706e246c1820eee"
Nov 03 02:29:04.495513 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:04.495494    5005 scope.go:122] "RemoveContainer" containerID="c2a8d3e4c498bc942aa45ad33c22fff919078c39d08f2dbc3706e246c1820eee"
Nov 03 02:29:04.495793 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:29:04.495771    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a8d3e4c498bc942aa45ad33c22fff919078c39d08f2dbc3706e246c1820eee\": container with ID starting with c2a8d3e4c498bc942aa45ad33c22fff919078c39d08f2dbc3706e246c1820eee not found: ID does not exist" containerID="c2a8d3e4c498bc942aa45ad33c22fff919078c39d08f2dbc3706e246c1820eee"
Nov 03 02:29:04.495852 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:04.495802    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a8d3e4c498bc942aa45ad33c22fff919078c39d08f2dbc3706e246c1820eee"} err="failed to get container status \"c2a8d3e4c498bc942aa45ad33c22fff919078c39d08f2dbc3706e246c1820eee\": rpc error: code = NotFound desc = could not find container \"c2a8d3e4c498bc942aa45ad33c22fff919078c39d08f2dbc3706e246c1820eee\": container with ID starting with c2a8d3e4c498bc942aa45ad33c22fff919078c39d08f2dbc3706e246c1820eee not found: ID does not exist"
Nov 03 02:29:04.638596 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:04.638535    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"]
Nov 03 02:29:04.641749 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:04.641728    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7f9d9d-66dc988879-dtm66"]
Nov 03 02:29:05.109791 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:05.109761    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6762613-dfe0-4e64-b920-6c479e144d55" path="/var/lib/kubelet/pods/b6762613-dfe0-4e64-b920-6c479e144d55/volumes"
Nov 03 02:29:07.474829 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:07.474797    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"
Nov 03 02:29:13.668730 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:13.668684    5005 scope.go:122] "RemoveContainer" containerID="c8736a3da247a0131f454e8f034a940fe9ab54fd38b089e48256bda3d3b36a5d"
Nov 03 02:29:30.716234 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:29:30.716148    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:29:58.666340 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:58.666299    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"]
Nov 03 02:29:58.666982 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:58.666965    5005 cpu_manager.go:414] "containerMap: removing container" podUID="b6762613-dfe0-4e64-b920-6c479e144d55" containerName="maas-api"
Nov 03 02:29:58.667032 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:58.666983    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6762613-dfe0-4e64-b920-6c479e144d55" containerName="maas-api"
Nov 03 02:29:58.667118 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:58.667106    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6762613-dfe0-4e64-b920-6c479e144d55" containerName="maas-api"
Nov 03 02:29:58.670825 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:58.670805    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"
Nov 03 02:29:58.719661 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:58.719616    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-711318-serving-cert\""
Nov 03 02:29:58.758737 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:58.758677    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-maas-api-tls\") pod \"maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6\" (UID: \"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"
Nov 03 02:29:58.758904 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:58.758865    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbz78\" (UniqueName: \"kubernetes.io/projected/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-kube-api-access-xbz78\") pod \"maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6\" (UID: \"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"
Nov 03 02:29:58.759019 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:58.759001    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-metrics-certs\") pod \"maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6\" (UID: \"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"
Nov 03 02:29:58.759074 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:58.759058    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6\" (UID: \"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"
Nov 03 02:29:58.768996 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:58.768973    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"]
Nov 03 02:29:58.860092 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:58.860058    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-maas-api-tls\") pod \"maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6\" (UID: \"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"
Nov 03 02:29:58.860291 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:58.860136    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbz78\" (UniqueName: \"kubernetes.io/projected/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-kube-api-access-xbz78\") pod \"maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6\" (UID: \"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"
Nov 03 02:29:58.860291 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:58.860189    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-metrics-certs\") pod \"maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6\" (UID: \"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"
Nov 03 02:29:58.860291 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:29:58.860217    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-serving-cert: secret "maas-api-e2e-praxis-b-711318-serving-cert" not found
Nov 03 02:29:58.860291 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:29:58.860288    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-maas-api-tls podName:2114664b-4364-4e6e-a1bc-cf54d2c4d8ce nodeName:}" failed. No retries permitted until 2026-11-03 02:29:59.360267065 +0000 UTC m=+2326.374508414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-maas-api-tls") pod "maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6" (UID: "2114664b-4364-4e6e-a1bc-cf54d2c4d8ce") : secret "maas-api-e2e-praxis-b-711318-serving-cert" not found
Nov 03 02:29:58.860514 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:58.860225    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6\" (UID: \"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"
Nov 03 02:29:58.860989 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:58.860970    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6\" (UID: \"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"
Nov 03 02:29:58.862841 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:58.862818    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-metrics-certs\") pod \"maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6\" (UID: \"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"
Nov 03 02:29:58.926621 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:58.926525    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbz78\" (UniqueName: \"kubernetes.io/projected/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-kube-api-access-xbz78\") pod \"maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6\" (UID: \"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"
Nov 03 02:29:59.367112 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:59.367075    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-maas-api-tls\") pod \"maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6\" (UID: \"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"
Nov 03 02:29:59.369632 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:59.369606    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-maas-api-tls\") pod \"maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6\" (UID: \"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"
Nov 03 02:29:59.583904 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:59.583852    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"
Nov 03 02:29:59.959940 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:29:59.959845    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"]
Nov 03 02:29:59.966108 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:29:59.966078    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2114664b_4364_4e6e_a1bc_cf54d2c4d8ce.slice/crio-227d619d8fb4de888514d2c7209f102a7f336c2deb7b86039faea3ef06523912 WatchSource:0}: Error finding container 227d619d8fb4de888514d2c7209f102a7f336c2deb7b86039faea3ef06523912: Status 404 returned error can't find the container with id 227d619d8fb4de888514d2c7209f102a7f336c2deb7b86039faea3ef06523912
Nov 03 02:30:00.715169 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:30:00.715119    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:30:00.847646 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:00.847606    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6" event={"ID":"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce","Type":"ContainerStarted","Data":"227d619d8fb4de888514d2c7209f102a7f336c2deb7b86039faea3ef06523912"}
Nov 03 02:30:01.856985 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:01.856945    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6" event={"ID":"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce","Type":"ContainerStarted","Data":"33ca0d811633c83b705725c63b9f94c75967490eb80a9b7eac6271557ac638b4"}
Nov 03 02:30:01.857433 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:01.857257    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"
Nov 03 02:30:01.964066 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:01.963984    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6" podStartSLOduration=2.92996422 podStartE2EDuration="3.963956798s" totalImagesPullingTime="1.033992578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:29:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:01.963429638 +0000 UTC m=+2328.977671006" watchObservedRunningTime="2026-11-03 02:30:01.963956798 +0000 UTC m=+2328.978198164"
Nov 03 02:30:07.869659 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:07.869621    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"
Nov 03 02:30:12.488064 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:12.488024    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"]
Nov 03 02:30:12.488539 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:12.488378    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk" podUID="7ebd4e3a-53a6-414a-94a2-726e9815b7aa" containerName="maas-api" containerID="cri-o://35b6cfe9faa938876ebc025a9d9064753382b0f4665ef11fd0728bbff68f2c23" gracePeriod=30
Nov 03 02:30:12.758802 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:12.758776    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"
Nov 03 02:30:12.833764 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:12.833725    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-metrics-certs\") pod \"7ebd4e3a-53a6-414a-94a2-726e9815b7aa\" (UID: \"7ebd4e3a-53a6-414a-94a2-726e9815b7aa\") "
Nov 03 02:30:12.833934 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:12.833812    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-maas-api-tls\") pod \"7ebd4e3a-53a6-414a-94a2-726e9815b7aa\" (UID: \"7ebd4e3a-53a6-414a-94a2-726e9815b7aa\") "
Nov 03 02:30:12.834049 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:12.834029    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-kube-api-access-m5sw8\" (UniqueName: \"kubernetes.io/projected/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-kube-api-access-m5sw8\") pod \"7ebd4e3a-53a6-414a-94a2-726e9815b7aa\" (UID: \"7ebd4e3a-53a6-414a-94a2-726e9815b7aa\") "
Nov 03 02:30:12.834101 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:12.834088    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-openshift-service-ca\") pod \"7ebd4e3a-53a6-414a-94a2-726e9815b7aa\" (UID: \"7ebd4e3a-53a6-414a-94a2-726e9815b7aa\") "
Nov 03 02:30:12.834557 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:12.834524    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-openshift-service-ca" pod "7ebd4e3a-53a6-414a-94a2-726e9815b7aa" (UID: "7ebd4e3a-53a6-414a-94a2-726e9815b7aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:30:12.835975 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:12.835950    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-maas-api-tls" pod "7ebd4e3a-53a6-414a-94a2-726e9815b7aa" (UID: "7ebd4e3a-53a6-414a-94a2-726e9815b7aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:30:12.836076 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:12.835972    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-metrics-certs" pod "7ebd4e3a-53a6-414a-94a2-726e9815b7aa" (UID: "7ebd4e3a-53a6-414a-94a2-726e9815b7aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:30:12.836076 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:12.836038    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-kube-api-access-m5sw8" pod "7ebd4e3a-53a6-414a-94a2-726e9815b7aa" (UID: "7ebd4e3a-53a6-414a-94a2-726e9815b7aa"). InnerVolumeSpecName "kube-api-access-m5sw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:30:12.928528 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:12.928493    5005 generic.go:356] "Generic (PLEG): container finished" podID="7ebd4e3a-53a6-414a-94a2-726e9815b7aa" containerID="35b6cfe9faa938876ebc025a9d9064753382b0f4665ef11fd0728bbff68f2c23" exitCode=0
Nov 03 02:30:12.928728 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:12.928541    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"
Nov 03 02:30:12.928728 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:12.928570    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk" event={"ID":"7ebd4e3a-53a6-414a-94a2-726e9815b7aa","Type":"ContainerDied","Data":"35b6cfe9faa938876ebc025a9d9064753382b0f4665ef11fd0728bbff68f2c23"}
Nov 03 02:30:12.928728 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:12.928660    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk" event={"ID":"7ebd4e3a-53a6-414a-94a2-726e9815b7aa","Type":"ContainerDied","Data":"c2d5af2cf99f0bf0a9c915364e5a4f22b78c013227d939823981b2350057dfb1"}
Nov 03 02:30:12.928728 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:12.928689    5005 scope.go:122] "RemoveContainer" containerID="35b6cfe9faa938876ebc025a9d9064753382b0f4665ef11fd0728bbff68f2c23"
Nov 03 02:30:12.935482 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:12.935455    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5sw8\" (UniqueName: \"kubernetes.io/projected/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-kube-api-access-m5sw8\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:12.935482 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:12.935481    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:12.935640 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:12.935494    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:12.935640 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:12.935503    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ebd4e3a-53a6-414a-94a2-726e9815b7aa-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:12.940651 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:12.940633    5005 scope.go:122] "RemoveContainer" containerID="35b6cfe9faa938876ebc025a9d9064753382b0f4665ef11fd0728bbff68f2c23"
Nov 03 02:30:12.941011 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:30:12.940993    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35b6cfe9faa938876ebc025a9d9064753382b0f4665ef11fd0728bbff68f2c23\": container with ID starting with 35b6cfe9faa938876ebc025a9d9064753382b0f4665ef11fd0728bbff68f2c23 not found: ID does not exist" containerID="35b6cfe9faa938876ebc025a9d9064753382b0f4665ef11fd0728bbff68f2c23"
Nov 03 02:30:12.941066 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:12.941021    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b6cfe9faa938876ebc025a9d9064753382b0f4665ef11fd0728bbff68f2c23"} err="failed to get container status \"35b6cfe9faa938876ebc025a9d9064753382b0f4665ef11fd0728bbff68f2c23\": rpc error: code = NotFound desc = could not find container \"35b6cfe9faa938876ebc025a9d9064753382b0f4665ef11fd0728bbff68f2c23\": container with ID starting with 35b6cfe9faa938876ebc025a9d9064753382b0f4665ef11fd0728bbff68f2c23 not found: ID does not exist"
Nov 03 02:30:13.082856 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:13.082774    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"]
Nov 03 02:30:13.085783 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:13.085753    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-56299c-68fd95bc9f-t2mjk"]
Nov 03 02:30:13.115137 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:13.115101    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ebd4e3a-53a6-414a-94a2-726e9815b7aa" path="/var/lib/kubelet/pods/7ebd4e3a-53a6-414a-94a2-726e9815b7aa/volumes"
Nov 03 02:30:20.389595 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.389557    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"]
Nov 03 02:30:20.390152 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.390138    5005 cpu_manager.go:414] "containerMap: removing container" podUID="7ebd4e3a-53a6-414a-94a2-726e9815b7aa" containerName="maas-api"
Nov 03 02:30:20.390201 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.390154    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ebd4e3a-53a6-414a-94a2-726e9815b7aa" containerName="maas-api"
Nov 03 02:30:20.390251 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.390242    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ebd4e3a-53a6-414a-94a2-726e9815b7aa" containerName="maas-api"
Nov 03 02:30:20.395963 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.395941    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.484945 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.484904    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"]
Nov 03 02:30:20.523482 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.523453    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f7d9bb-openshift-default-dockercfg-dp5lm\""
Nov 03 02:30:20.529571 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.529543    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hft4b\" (UniqueName: \"kubernetes.io/projected/cefe8bab-136b-428e-8173-c6c89259fe5f-kube-api-access-hft4b\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.529714 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.529596    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.529714 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.529683    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.529818 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.529757    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefe8bab-136b-428e-8173-c6c89259fe5f-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.529866 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.529847    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.529907 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.529895    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.529991 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.529978    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.530034 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.530021    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.530115 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.530103    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.631319 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.631282    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.631500 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.631395    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hft4b\" (UniqueName: \"kubernetes.io/projected/cefe8bab-136b-428e-8173-c6c89259fe5f-kube-api-access-hft4b\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.631500 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.631437    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.631500 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.631485    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.631676 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.631509    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefe8bab-136b-428e-8173-c6c89259fe5f-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.631676 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.631547    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.631676 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.631618    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.631874 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.631677    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.631874 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.631761    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.631991 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.631956    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.632134 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.631993    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.632134 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.632022    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.632447 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.632423    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.632631 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.632605    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefe8bab-136b-428e-8173-c6c89259fe5f-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.634116 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.634094    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.634343 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.634318    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.684416 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.684345    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.684564 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.684543    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hft4b\" (UniqueName: \"kubernetes.io/projected/cefe8bab-136b-428e-8173-c6c89259fe5f-kube-api-access-hft4b\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:20.713952 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:20.713924    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:21.145457 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:21.145419    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"]
Nov 03 02:30:21.150087 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:30:21.150059    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcefe8bab_136b_428e_8173_c6c89259fe5f.slice/crio-71b1e1045b0c2ce703bb513d3499aeb0163cb02b3177f72f74c319e9845730e3 WatchSource:0}: Error finding container 71b1e1045b0c2ce703bb513d3499aeb0163cb02b3177f72f74c319e9845730e3: Status 404 returned error can't find the container with id 71b1e1045b0c2ce703bb513d3499aeb0163cb02b3177f72f74c319e9845730e3
Nov 03 02:30:21.152463 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:21.152428    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:30:21.152545 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:21.152505    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:30:21.152545 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:21.152539    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:30:21.986958 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:21.986912    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2" event={"ID":"cefe8bab-136b-428e-8173-c6c89259fe5f","Type":"ContainerStarted","Data":"5db5d66bfc9ec7d472e85820b52b3184a21300f4ff07ce393f7514c9f5696e49"}
Nov 03 02:30:21.987418 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:21.987007    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2" event={"ID":"cefe8bab-136b-428e-8173-c6c89259fe5f","Type":"ContainerStarted","Data":"71b1e1045b0c2ce703bb513d3499aeb0163cb02b3177f72f74c319e9845730e3"}
Nov 03 02:30:22.720551 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:22.719762    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:22.720919 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:22.720894    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 02:30:22.721096 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:22.721073    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2" podUID="cefe8bab-136b-428e-8173-c6c89259fe5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 02:30:23.720719 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:23.716043    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 02:30:23.720719 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:23.716110    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2" podUID="cefe8bab-136b-428e-8173-c6c89259fe5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 02:30:24.420160 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:24.420094    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2" podStartSLOduration=4.420072497 podStartE2EDuration="4.420072497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:22.150864293 +0000 UTC m=+2349.165105672" watchObservedRunningTime="2026-11-03 02:30:24.420072497 +0000 UTC m=+2351.434313864"
Nov 03 02:30:24.425798 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:24.424821    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"]
Nov 03 02:30:24.425798 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:24.425156    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh" podUID="7d6ce282-686b-432c-bb78-7361aecf6a05" containerName="istio-proxy" containerID="cri-o://e485576da0fb9c8e9165dffb4cbb13f9f2c0aa538dfac85ea87b19822dcd78b5" gracePeriod=30
Nov 03 02:30:24.716769 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:24.716069    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 02:30:24.716769 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:24.716130    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2" podUID="cefe8bab-136b-428e-8173-c6c89259fe5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 02:30:25.260832 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:25.260793    5005 patch_prober.go:29] interesting pod/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 02:30:25.261268 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:25.260858    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh" podUID="7d6ce282-686b-432c-bb78-7361aecf6a05" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 02:30:25.723153 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:25.723057    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 02:30:25.723153 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:25.723116    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2" podUID="cefe8bab-136b-428e-8173-c6c89259fe5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 02:30:26.720717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:26.719117    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 02:30:26.720717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:26.719199    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2" podUID="cefe8bab-136b-428e-8173-c6c89259fe5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 02:30:27.714626 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:27.714587    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 03 02:30:27.714844 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:27.714662    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2" podUID="cefe8bab-136b-428e-8173-c6c89259fe5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 03 02:30:29.132976 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.132940    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:29.133431 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.133350    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:29.134116 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.134096    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:30:29.701313 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.701253    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:30:29.871545 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.871509    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-podinfo\") pod \"7d6ce282-686b-432c-bb78-7361aecf6a05\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") "
Nov 03 02:30:29.871759 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.871564    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-workload-certs\") pod \"7d6ce282-686b-432c-bb78-7361aecf6a05\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") "
Nov 03 02:30:29.871759 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.871607    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-token\" (UniqueName: \"kubernetes.io/projected/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-token\") pod \"7d6ce282-686b-432c-bb78-7361aecf6a05\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") "
Nov 03 02:30:29.871759 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.871662    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-envoy\") pod \"7d6ce282-686b-432c-bb78-7361aecf6a05\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") "
Nov 03 02:30:29.872021 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.871763    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-data\") pod \"7d6ce282-686b-432c-bb78-7361aecf6a05\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") "
Nov 03 02:30:29.872021 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.871859    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-workload-certs" pod "7d6ce282-686b-432c-bb78-7361aecf6a05" (UID: "7d6ce282-686b-432c-bb78-7361aecf6a05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:30:29.872021 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.871896    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d6ce282-686b-432c-bb78-7361aecf6a05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6ce282-686b-432c-bb78-7361aecf6a05-istiod-ca-cert\") pod \"7d6ce282-686b-432c-bb78-7361aecf6a05\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") "
Nov 03 02:30:29.872021 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.871944    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-workload-socket\") pod \"7d6ce282-686b-432c-bb78-7361aecf6a05\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") "
Nov 03 02:30:29.872021 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.872005    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6ce282-686b-432c-bb78-7361aecf6a05-kube-api-access-dgzqv\" (UniqueName: \"kubernetes.io/projected/7d6ce282-686b-432c-bb78-7361aecf6a05-kube-api-access-dgzqv\") pod \"7d6ce282-686b-432c-bb78-7361aecf6a05\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") "
Nov 03 02:30:29.872304 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.872069    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-credential-socket\") pod \"7d6ce282-686b-432c-bb78-7361aecf6a05\" (UID: \"7d6ce282-686b-432c-bb78-7361aecf6a05\") "
Nov 03 02:30:29.872304 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.872112    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-data" pod "7d6ce282-686b-432c-bb78-7361aecf6a05" (UID: "7d6ce282-686b-432c-bb78-7361aecf6a05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:30:29.872407 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.872305    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-workload-socket" pod "7d6ce282-686b-432c-bb78-7361aecf6a05" (UID: "7d6ce282-686b-432c-bb78-7361aecf6a05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:30:29.872407 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.872314    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6ce282-686b-432c-bb78-7361aecf6a05-istiod-ca-cert" pod "7d6ce282-686b-432c-bb78-7361aecf6a05" (UID: "7d6ce282-686b-432c-bb78-7361aecf6a05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:30:29.872887 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.872512    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-credential-socket" pod "7d6ce282-686b-432c-bb78-7361aecf6a05" (UID: "7d6ce282-686b-432c-bb78-7361aecf6a05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:30:29.872887 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.872757    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:29.872887 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.872784    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6ce282-686b-432c-bb78-7361aecf6a05-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:29.872887 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.872803    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:29.872887 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.872818    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:29.872887 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.872831    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:29.874606 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.874576    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-envoy" pod "7d6ce282-686b-432c-bb78-7361aecf6a05" (UID: "7d6ce282-686b-432c-bb78-7361aecf6a05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:30:29.874758 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.874678    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6ce282-686b-432c-bb78-7361aecf6a05-kube-api-access-dgzqv" pod "7d6ce282-686b-432c-bb78-7361aecf6a05" (UID: "7d6ce282-686b-432c-bb78-7361aecf6a05"). InnerVolumeSpecName "kube-api-access-dgzqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:30:29.874758 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.874730    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-podinfo" pod "7d6ce282-686b-432c-bb78-7361aecf6a05" (UID: "7d6ce282-686b-432c-bb78-7361aecf6a05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:30:29.874861 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.874774    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-token" pod "7d6ce282-686b-432c-bb78-7361aecf6a05" (UID: "7d6ce282-686b-432c-bb78-7361aecf6a05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:30:29.973844 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.973745    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:29.973844 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.973780    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:29.973844 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.973792    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6ce282-686b-432c-bb78-7361aecf6a05-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:29.973844 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:29.973805    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgzqv\" (UniqueName: \"kubernetes.io/projected/7d6ce282-686b-432c-bb78-7361aecf6a05-kube-api-access-dgzqv\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:30.076064 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:30.076030    5005 generic.go:356] "Generic (PLEG): container finished" podID="7d6ce282-686b-432c-bb78-7361aecf6a05" containerID="e485576da0fb9c8e9165dffb4cbb13f9f2c0aa538dfac85ea87b19822dcd78b5" exitCode=0
Nov 03 02:30:30.076257 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:30.076077    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"
Nov 03 02:30:30.076257 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:30.076118    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh" event={"ID":"7d6ce282-686b-432c-bb78-7361aecf6a05","Type":"ContainerDied","Data":"e485576da0fb9c8e9165dffb4cbb13f9f2c0aa538dfac85ea87b19822dcd78b5"}
Nov 03 02:30:30.076257 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:30.076174    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh" event={"ID":"7d6ce282-686b-432c-bb78-7361aecf6a05","Type":"ContainerDied","Data":"6d38fce96a0016bea98d055677aaea324384cd0677c9d436b77135ad88ab3dd8"}
Nov 03 02:30:30.076257 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:30.076203    5005 scope.go:122] "RemoveContainer" containerID="e485576da0fb9c8e9165dffb4cbb13f9f2c0aa538dfac85ea87b19822dcd78b5"
Nov 03 02:30:30.087104 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:30.087074    5005 scope.go:122] "RemoveContainer" containerID="e485576da0fb9c8e9165dffb4cbb13f9f2c0aa538dfac85ea87b19822dcd78b5"
Nov 03 02:30:30.087392 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:30:30.087356    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e485576da0fb9c8e9165dffb4cbb13f9f2c0aa538dfac85ea87b19822dcd78b5\": container with ID starting with e485576da0fb9c8e9165dffb4cbb13f9f2c0aa538dfac85ea87b19822dcd78b5 not found: ID does not exist" containerID="e485576da0fb9c8e9165dffb4cbb13f9f2c0aa538dfac85ea87b19822dcd78b5"
Nov 03 02:30:30.087458 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:30.087388    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e485576da0fb9c8e9165dffb4cbb13f9f2c0aa538dfac85ea87b19822dcd78b5"} err="failed to get container status \"e485576da0fb9c8e9165dffb4cbb13f9f2c0aa538dfac85ea87b19822dcd78b5\": rpc error: code = NotFound desc = could not find container \"e485576da0fb9c8e9165dffb4cbb13f9f2c0aa538dfac85ea87b19822dcd78b5\": container with ID starting with e485576da0fb9c8e9165dffb4cbb13f9f2c0aa538dfac85ea87b19822dcd78b5 not found: ID does not exist"
Nov 03 02:30:30.230428 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:30.230351    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"]
Nov 03 02:30:30.235468 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:30.235441    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7d758-openshift-default-6586c5845c-bxxxh"]
Nov 03 02:30:30.438972 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:30.438928    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"]
Nov 03 02:30:30.439229 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:30.439199    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs" podUID="6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8" containerName="istio-proxy" containerID="cri-o://b2bf49fc1c6762b9886c9bcb565760a84e9ce2cb8fff6933c7b2ae98a3544692" gracePeriod=30
Nov 03 02:30:30.716792 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:30:30.716756    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:30:31.110645 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:31.110611    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d6ce282-686b-432c-bb78-7361aecf6a05" path="/var/lib/kubelet/pods/7d6ce282-686b-432c-bb78-7361aecf6a05/volumes"
Nov 03 02:30:33.785910 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:33.785875    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"]
Nov 03 02:30:33.786662 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:33.786639    5005 cpu_manager.go:414] "containerMap: removing container" podUID="7d6ce282-686b-432c-bb78-7361aecf6a05" containerName="istio-proxy"
Nov 03 02:30:33.786789 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:33.786665    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d6ce282-686b-432c-bb78-7361aecf6a05" containerName="istio-proxy"
Nov 03 02:30:33.786864 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:33.786849    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d6ce282-686b-432c-bb78-7361aecf6a05" containerName="istio-proxy"
Nov 03 02:30:33.790596 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:33.790571    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"
Nov 03 02:30:33.885865 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:33.885833    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f9bff9-serving-cert\""
Nov 03 02:30:33.889414 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:33.889388    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"]
Nov 03 02:30:33.923382 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:33.923332    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j\" (UID: \"ae2fd3df-1c03-4b71-8b58-0b7532fbd668\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"
Nov 03 02:30:33.923550 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:33.923505    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-925rq\" (UniqueName: \"kubernetes.io/projected/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-kube-api-access-925rq\") pod \"maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j\" (UID: \"ae2fd3df-1c03-4b71-8b58-0b7532fbd668\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"
Nov 03 02:30:33.923776 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:33.923754    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j\" (UID: \"ae2fd3df-1c03-4b71-8b58-0b7532fbd668\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"
Nov 03 02:30:33.923882 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:33.923863    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-metrics-certs\") pod \"maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j\" (UID: \"ae2fd3df-1c03-4b71-8b58-0b7532fbd668\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"
Nov 03 02:30:34.025725 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:34.025646    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j\" (UID: \"ae2fd3df-1c03-4b71-8b58-0b7532fbd668\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"
Nov 03 02:30:34.029793 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:34.025758    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-metrics-certs\") pod \"maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j\" (UID: \"ae2fd3df-1c03-4b71-8b58-0b7532fbd668\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"
Nov 03 02:30:34.029793 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:34.025818    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j\" (UID: \"ae2fd3df-1c03-4b71-8b58-0b7532fbd668\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"
Nov 03 02:30:34.029793 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:30:34.025834    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-serving-cert: secret "maas-api-e2e-ipp-b-f9bff9-serving-cert" not found
Nov 03 02:30:34.029793 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:34.025897    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-925rq\" (UniqueName: \"kubernetes.io/projected/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-kube-api-access-925rq\") pod \"maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j\" (UID: \"ae2fd3df-1c03-4b71-8b58-0b7532fbd668\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"
Nov 03 02:30:34.029793 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:30:34.025921    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-maas-api-tls podName:ae2fd3df-1c03-4b71-8b58-0b7532fbd668 nodeName:}" failed. No retries permitted until 2026-11-03 02:30:34.525898944 +0000 UTC m=+2361.540140311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-maas-api-tls") pod "maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j" (UID: "ae2fd3df-1c03-4b71-8b58-0b7532fbd668") : secret "maas-api-e2e-ipp-b-f9bff9-serving-cert" not found
Nov 03 02:30:34.029793 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:34.027070    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j\" (UID: \"ae2fd3df-1c03-4b71-8b58-0b7532fbd668\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"
Nov 03 02:30:34.037504 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:34.037362    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-metrics-certs\") pod \"maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j\" (UID: \"ae2fd3df-1c03-4b71-8b58-0b7532fbd668\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"
Nov 03 02:30:34.079608 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:34.079579    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-925rq\" (UniqueName: \"kubernetes.io/projected/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-kube-api-access-925rq\") pod \"maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j\" (UID: \"ae2fd3df-1c03-4b71-8b58-0b7532fbd668\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"
Nov 03 02:30:34.531757 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:34.531725    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j\" (UID: \"ae2fd3df-1c03-4b71-8b58-0b7532fbd668\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"
Nov 03 02:30:34.534159 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:34.534136    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j\" (UID: \"ae2fd3df-1c03-4b71-8b58-0b7532fbd668\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"
Nov 03 02:30:34.703643 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:34.703606    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"
Nov 03 02:30:35.116988 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.116904    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"]
Nov 03 02:30:35.121587 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:30:35.121528    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2fd3df_1c03_4b71_8b58_0b7532fbd668.slice/crio-dc0616928a193f96e9397224fa72af5ce48fe114728ef39d00ad392c25e71f28 WatchSource:0}: Error finding container dc0616928a193f96e9397224fa72af5ce48fe114728ef39d00ad392c25e71f28: Status 404 returned error can't find the container with id dc0616928a193f96e9397224fa72af5ce48fe114728ef39d00ad392c25e71f28
Nov 03 02:30:35.124521 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.124499    5005 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 02:30:35.715519 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.715456    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:30:35.743136 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.743103    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istiod-ca-cert\") pod \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") "
Nov 03 02:30:35.743136 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.743139    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-token\" (UniqueName: \"kubernetes.io/projected/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-token\") pod \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") "
Nov 03 02:30:35.743341 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.743199    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-kube-api-access-p9w27\" (UniqueName: \"kubernetes.io/projected/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-kube-api-access-p9w27\") pod \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") "
Nov 03 02:30:35.743341 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.743234    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-envoy\") pod \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") "
Nov 03 02:30:35.743341 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.743280    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-workload-certs\") pod \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") "
Nov 03 02:30:35.743506 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.743347    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-credential-socket\") pod \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") "
Nov 03 02:30:35.743506 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.743374    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-workload-socket\") pod \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") "
Nov 03 02:30:35.743506 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.743454    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-data\") pod \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") "
Nov 03 02:30:35.743506 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.743485    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istiod-ca-cert" pod "6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8" (UID: "6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:30:35.743730 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.743551    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-podinfo\") pod \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\" (UID: \"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8\") "
Nov 03 02:30:35.743730 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.743705    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-workload-certs" pod "6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8" (UID: "6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:30:35.743851 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.743731    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-credential-socket" pod "6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8" (UID: "6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:30:35.743851 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.743811    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-data" pod "6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8" (UID: "6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:30:35.743949 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.743894    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-workload-socket" pod "6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8" (UID: "6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:30:35.744085 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.744066    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:35.744127 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.744093    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:35.744127 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.744109    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:35.744127 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.744123    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:35.744237 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.744137    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:35.745577 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.745541    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-token" pod "6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8" (UID: "6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:30:35.745672 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.745636    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-podinfo" pod "6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8" (UID: "6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:30:35.745672 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.745655    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-kube-api-access-p9w27" pod "6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8" (UID: "6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8"). InnerVolumeSpecName "kube-api-access-p9w27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:30:35.745897 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.745876    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-envoy" pod "6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8" (UID: "6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:30:35.845537 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.845499    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:35.845537 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.845531    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:35.845537 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.845541    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9w27\" (UniqueName: \"kubernetes.io/projected/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-kube-api-access-p9w27\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:35.845822 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:35.845552    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:36.121578 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:36.121524    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j" event={"ID":"ae2fd3df-1c03-4b71-8b58-0b7532fbd668","Type":"ContainerStarted","Data":"368816be067415171fce80b9e6ff08b47112a7ef03178e444aed34e4cdae287a"}
Nov 03 02:30:36.122072 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:36.121610    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j" event={"ID":"ae2fd3df-1c03-4b71-8b58-0b7532fbd668","Type":"ContainerStarted","Data":"dc0616928a193f96e9397224fa72af5ce48fe114728ef39d00ad392c25e71f28"}
Nov 03 02:30:36.122072 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:36.121857    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"
Nov 03 02:30:36.123522 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:36.123497    5005 generic.go:356] "Generic (PLEG): container finished" podID="6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8" containerID="b2bf49fc1c6762b9886c9bcb565760a84e9ce2cb8fff6933c7b2ae98a3544692" exitCode=0
Nov 03 02:30:36.123731 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:36.123529    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs" event={"ID":"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8","Type":"ContainerDied","Data":"b2bf49fc1c6762b9886c9bcb565760a84e9ce2cb8fff6933c7b2ae98a3544692"}
Nov 03 02:30:36.123880 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:36.123755    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs" event={"ID":"6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8","Type":"ContainerDied","Data":"624f6fb0f13d6885f56b9c08589b734438774e4c3b0aaa6141f48b8eb1a57be2"}
Nov 03 02:30:36.123880 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:36.123578    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"
Nov 03 02:30:36.124002 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:36.123790    5005 scope.go:122] "RemoveContainer" containerID="b2bf49fc1c6762b9886c9bcb565760a84e9ce2cb8fff6933c7b2ae98a3544692"
Nov 03 02:30:36.137347 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:36.137326    5005 scope.go:122] "RemoveContainer" containerID="b2bf49fc1c6762b9886c9bcb565760a84e9ce2cb8fff6933c7b2ae98a3544692"
Nov 03 02:30:36.137634 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:30:36.137610    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2bf49fc1c6762b9886c9bcb565760a84e9ce2cb8fff6933c7b2ae98a3544692\": container with ID starting with b2bf49fc1c6762b9886c9bcb565760a84e9ce2cb8fff6933c7b2ae98a3544692 not found: ID does not exist" containerID="b2bf49fc1c6762b9886c9bcb565760a84e9ce2cb8fff6933c7b2ae98a3544692"
Nov 03 02:30:36.137720 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:36.137642    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2bf49fc1c6762b9886c9bcb565760a84e9ce2cb8fff6933c7b2ae98a3544692"} err="failed to get container status \"b2bf49fc1c6762b9886c9bcb565760a84e9ce2cb8fff6933c7b2ae98a3544692\": rpc error: code = NotFound desc = could not find container \"b2bf49fc1c6762b9886c9bcb565760a84e9ce2cb8fff6933c7b2ae98a3544692\": container with ID starting with b2bf49fc1c6762b9886c9bcb565760a84e9ce2cb8fff6933c7b2ae98a3544692 not found: ID does not exist"
Nov 03 02:30:36.229228 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:36.229165    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j" podStartSLOduration=2.363084839 podStartE2EDuration="3.229147864s" totalImagesPullingTime="866.063025ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:36.2281316 +0000 UTC m=+2363.242372967" watchObservedRunningTime="2026-11-03 02:30:36.229147864 +0000 UTC m=+2363.243389231"
Nov 03 02:30:36.332944 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:36.332907    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"]
Nov 03 02:30:36.346046 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:36.340491    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ml5cs"]
Nov 03 02:30:37.109512 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:37.109477    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8" path="/var/lib/kubelet/pods/6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8/volumes"
Nov 03 02:30:42.143112 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:42.143077    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"
Nov 03 02:30:55.359358 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.359278    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"]
Nov 03 02:30:55.362185 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.359848    5005 cpu_manager.go:414] "containerMap: removing container" podUID="6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8" containerName="istio-proxy"
Nov 03 02:30:55.362185 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.359860    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8" containerName="istio-proxy"
Nov 03 02:30:55.362185 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.359954    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab48a26-11f9-4f6e-b0ab-8cbe7d34d0b8" containerName="istio-proxy"
Nov 03 02:30:55.363507 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.363483    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.411803 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.411775    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f9bff9-openshift-default-dockercfg-nn2lt\""
Nov 03 02:30:55.495485 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.495448    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.495837 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.495805    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.495964 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.495947    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.496036 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.496022    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05470e4a-2ba0-4978-bd4f-99040bd6119d-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.496073 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.496059    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.496169 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.496148    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.496256 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.496241    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.496292 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.496266    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c8nr\" (UniqueName: \"kubernetes.io/projected/05470e4a-2ba0-4978-bd4f-99040bd6119d-kube-api-access-9c8nr\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.496349 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.496334    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.554454 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.554420    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"]
Nov 03 02:30:55.597186 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.597146    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.597379 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.597194    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c8nr\" (UniqueName: \"kubernetes.io/projected/05470e4a-2ba0-4978-bd4f-99040bd6119d-kube-api-access-9c8nr\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.597379 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.597245    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.597379 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.597272    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.597379 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.597307    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.597610 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.597412    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.597610 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.597481    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05470e4a-2ba0-4978-bd4f-99040bd6119d-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.597610 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.597516    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.597610 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.597563    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.597861 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.597605    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.597861 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.597655    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.598029 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.597877    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.598029 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.597981    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.598474 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.598451    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05470e4a-2ba0-4978-bd4f-99040bd6119d-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.599633 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.599605    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.600668 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.600642    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.650208 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.650142    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c8nr\" (UniqueName: \"kubernetes.io/projected/05470e4a-2ba0-4978-bd4f-99040bd6119d-kube-api-access-9c8nr\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.650359 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.650342    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:55.677396 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:55.677376    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:56.111612 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:56.111581    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"]
Nov 03 02:30:56.117838 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:30:56.117801    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05470e4a_2ba0_4978_bd4f_99040bd6119d.slice/crio-5922836c01b8ae51eafc870044c43d5ff249a2a8dc0262a3ba79aca346a7838f WatchSource:0}: Error finding container 5922836c01b8ae51eafc870044c43d5ff249a2a8dc0262a3ba79aca346a7838f: Status 404 returned error can't find the container with id 5922836c01b8ae51eafc870044c43d5ff249a2a8dc0262a3ba79aca346a7838f
Nov 03 02:30:56.120208 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:56.120172    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:30:56.120275 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:56.120245    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:30:56.120311 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:56.120274    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:30:56.249042 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:56.249004    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6" event={"ID":"05470e4a-2ba0-4978-bd4f-99040bd6119d","Type":"ContainerStarted","Data":"80db72b5ba940dc209193d3803a9f2ae5bddd69cb83366b41845752a925885c2"}
Nov 03 02:30:56.249234 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:56.249098    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6" event={"ID":"05470e4a-2ba0-4978-bd4f-99040bd6119d","Type":"ContainerStarted","Data":"5922836c01b8ae51eafc870044c43d5ff249a2a8dc0262a3ba79aca346a7838f"}
Nov 03 02:30:56.359721 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:56.359645    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6" podStartSLOduration=1.359629204 podStartE2EDuration="1.359629204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:56.358778582 +0000 UTC m=+2383.373019948" watchObservedRunningTime="2026-11-03 02:30:56.359629204 +0000 UTC m=+2383.373870569"
Nov 03 02:30:56.677834 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:56.677796    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:30:57.679561 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:57.679529    5005 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 02:30:57.679991 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:57.679578    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6" podUID="05470e4a-2ba0-4978-bd4f-99040bd6119d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 02:30:58.678084 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:58.678048    5005 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 02:30:58.678273 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:58.678107    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6" podUID="05470e4a-2ba0-4978-bd4f-99040bd6119d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 02:30:59.678834 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:59.678800    5005 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 02:30:59.679343 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:30:59.679314    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6" podUID="05470e4a-2ba0-4978-bd4f-99040bd6119d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 02:31:00.678620 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:00.678586    5005 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 02:31:00.678888 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:00.678637    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6" podUID="05470e4a-2ba0-4978-bd4f-99040bd6119d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 02:31:00.737383 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:31:00.737150    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:31:01.678571 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:01.678529    5005 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 02:31:01.678772 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:01.678594    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6" podUID="05470e4a-2ba0-4978-bd4f-99040bd6119d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 02:31:02.683014 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:02.682979    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:31:02.683452 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:02.683439    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:31:02.684136 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:02.684118    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:31:13.442058 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:13.442026    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:31:13.453377 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:13.453350    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:31:15.874218 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:15.874182    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"]
Nov 03 02:31:15.874660 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:15.874419    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh" podUID="a601098f-bdb7-402c-a5d2-4cfe04ff1ca4" containerName="maas-api" containerID="cri-o://21e0cd42a67a111a0362d06eaaab957efe529f8e5bd965edb6a35692c2a0106a" gracePeriod=30
Nov 03 02:31:16.181469 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:16.177968    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"
Nov 03 02:31:16.284648 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:16.283588    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-openshift-service-ca\") pod \"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4\" (UID: \"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4\") "
Nov 03 02:31:16.284648 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:16.283657    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-metrics-certs\") pod \"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4\" (UID: \"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4\") "
Nov 03 02:31:16.284648 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:16.283730    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-maas-api-tls\") pod \"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4\" (UID: \"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4\") "
Nov 03 02:31:16.284648 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:16.283802    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-kube-api-access-5npf2\" (UniqueName: \"kubernetes.io/projected/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-kube-api-access-5npf2\") pod \"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4\" (UID: \"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4\") "
Nov 03 02:31:16.286546 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:16.285863    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-openshift-service-ca" pod "a601098f-bdb7-402c-a5d2-4cfe04ff1ca4" (UID: "a601098f-bdb7-402c-a5d2-4cfe04ff1ca4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:31:16.296637 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:16.296593    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-kube-api-access-5npf2" pod "a601098f-bdb7-402c-a5d2-4cfe04ff1ca4" (UID: "a601098f-bdb7-402c-a5d2-4cfe04ff1ca4"). InnerVolumeSpecName "kube-api-access-5npf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:31:16.319371 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:16.318839    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-metrics-certs" pod "a601098f-bdb7-402c-a5d2-4cfe04ff1ca4" (UID: "a601098f-bdb7-402c-a5d2-4cfe04ff1ca4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:31:16.346498 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:16.346411    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-maas-api-tls" pod "a601098f-bdb7-402c-a5d2-4cfe04ff1ca4" (UID: "a601098f-bdb7-402c-a5d2-4cfe04ff1ca4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:31:16.386054 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:16.385972    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:16.386054 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:16.386002    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:16.386054 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:16.386012    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:16.386054 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:16.386020    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5npf2\" (UniqueName: \"kubernetes.io/projected/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4-kube-api-access-5npf2\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:16.388238 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:16.388210    5005 generic.go:356] "Generic (PLEG): container finished" podID="a601098f-bdb7-402c-a5d2-4cfe04ff1ca4" containerID="21e0cd42a67a111a0362d06eaaab957efe529f8e5bd965edb6a35692c2a0106a" exitCode=0
Nov 03 02:31:16.388365 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:16.388263    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"
Nov 03 02:31:16.388365 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:16.388256    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh" event={"ID":"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4","Type":"ContainerDied","Data":"21e0cd42a67a111a0362d06eaaab957efe529f8e5bd965edb6a35692c2a0106a"}
Nov 03 02:31:16.388489 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:16.388392    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh" event={"ID":"a601098f-bdb7-402c-a5d2-4cfe04ff1ca4","Type":"ContainerDied","Data":"e242a2781ab2694505aeb362d6b272eaba9e0e0391616412468ec93cb16b08e4"}
Nov 03 02:31:16.388489 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:16.388430    5005 scope.go:122] "RemoveContainer" containerID="21e0cd42a67a111a0362d06eaaab957efe529f8e5bd965edb6a35692c2a0106a"
Nov 03 02:31:16.399473 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:16.399448    5005 scope.go:122] "RemoveContainer" containerID="21e0cd42a67a111a0362d06eaaab957efe529f8e5bd965edb6a35692c2a0106a"
Nov 03 02:31:16.399808 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:31:16.399784    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e0cd42a67a111a0362d06eaaab957efe529f8e5bd965edb6a35692c2a0106a\": container with ID starting with 21e0cd42a67a111a0362d06eaaab957efe529f8e5bd965edb6a35692c2a0106a not found: ID does not exist" containerID="21e0cd42a67a111a0362d06eaaab957efe529f8e5bd965edb6a35692c2a0106a"
Nov 03 02:31:16.399869 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:16.399822    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e0cd42a67a111a0362d06eaaab957efe529f8e5bd965edb6a35692c2a0106a"} err="failed to get container status \"21e0cd42a67a111a0362d06eaaab957efe529f8e5bd965edb6a35692c2a0106a\": rpc error: code = NotFound desc = could not find container \"21e0cd42a67a111a0362d06eaaab957efe529f8e5bd965edb6a35692c2a0106a\": container with ID starting with 21e0cd42a67a111a0362d06eaaab957efe529f8e5bd965edb6a35692c2a0106a not found: ID does not exist"
Nov 03 02:31:16.542197 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:16.542157    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"]
Nov 03 02:31:16.545383 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:16.545357    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-496177-5d579bd8b-gn4hh"]
Nov 03 02:31:17.110375 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:17.110338    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a601098f-bdb7-402c-a5d2-4cfe04ff1ca4" path="/var/lib/kubelet/pods/a601098f-bdb7-402c-a5d2-4cfe04ff1ca4/volumes"
Nov 03 02:31:20.865901 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:20.865868    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"]
Nov 03 02:31:20.866475 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:20.866460    5005 cpu_manager.go:414] "containerMap: removing container" podUID="a601098f-bdb7-402c-a5d2-4cfe04ff1ca4" containerName="maas-api"
Nov 03 02:31:20.866519 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:20.866477    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a601098f-bdb7-402c-a5d2-4cfe04ff1ca4" containerName="maas-api"
Nov 03 02:31:20.866636 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:20.866626    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="a601098f-bdb7-402c-a5d2-4cfe04ff1ca4" containerName="maas-api"
Nov 03 02:31:20.871537 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:20.871512    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.045935 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.045899    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.045935 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.045935    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hblxw\" (UniqueName: \"kubernetes.io/projected/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-kube-api-access-hblxw\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.046203 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.045963    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.046203 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.046056    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.046309 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.046220    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.046309 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.046259    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.046415 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.046356    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.046496 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.046455    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.046555 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.046543    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.059595 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.059559    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"]
Nov 03 02:31:21.148090 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.147989    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.148090 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.148048    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.148090 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.148082    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.148393 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.148117    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.148393 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.148148    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.148393 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.148180    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.148393 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.148207    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hblxw\" (UniqueName: \"kubernetes.io/projected/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-kube-api-access-hblxw\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.148393 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.148242    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.148393 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.148275    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.148806 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.148500    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.148806 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.148595    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.148806 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.148679    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.148806 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.148777    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.149070 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.149047    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.150665 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.150637    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.151060 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.151013    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.202167 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.202130    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.202308 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.202134    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hblxw\" (UniqueName: \"kubernetes.io/projected/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-kube-api-access-hblxw\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.486422 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.486328    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:21.986764 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.986722    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"]
Nov 03 02:31:21.992680 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:31:21.992649    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecdc91da_4a69_4e9a_b46c_12d9cf909aa2.slice/crio-9ae55b5d3e511540cff286d4708e1191389038858bec738ede46440f71ecb0ee WatchSource:0}: Error finding container 9ae55b5d3e511540cff286d4708e1191389038858bec738ede46440f71ecb0ee: Status 404 returned error can't find the container with id 9ae55b5d3e511540cff286d4708e1191389038858bec738ede46440f71ecb0ee
Nov 03 02:31:21.995172 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.995135    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:31:21.995241 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.995209    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:31:21.995273 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:21.995238    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:31:22.433661 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:22.433616    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr" event={"ID":"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2","Type":"ContainerStarted","Data":"517e2513e9d6e615828d0d8e436e38a27780ba50b6c6acb43e349c0857e159eb"}
Nov 03 02:31:22.433898 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:22.433681    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr" event={"ID":"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2","Type":"ContainerStarted","Data":"9ae55b5d3e511540cff286d4708e1191389038858bec738ede46440f71ecb0ee"}
Nov 03 02:31:22.486902 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:22.486867    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:22.594094 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:22.594034    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr" podStartSLOduration=2.594018122 podStartE2EDuration="2.594018122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:31:22.591081579 +0000 UTC m=+2409.605322946" watchObservedRunningTime="2026-11-03 02:31:22.594018122 +0000 UTC m=+2409.608259488"
Nov 03 02:31:23.489069 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:23.489034    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 02:31:23.489487 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:23.489088    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr" podUID="ecdc91da-4a69-4e9a-b46c-12d9cf909aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 02:31:24.486865 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:24.486830    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 02:31:24.487047 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:24.486894    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr" podUID="ecdc91da-4a69-4e9a-b46c-12d9cf909aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 02:31:25.487467 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.487431    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 02:31:25.487894 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.487487    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr" podUID="ecdc91da-4a69-4e9a-b46c-12d9cf909aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 02:31:25.590740 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.590704    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"]
Nov 03 02:31:25.594972 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.594951    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.599350 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.598152    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.599350 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.598208    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.599350 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.598248    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.599350 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.598277    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a060f3-9db2-4020-bacf-6b04c0941a3a-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.599350 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.598315    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.599350 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.598375    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.599350 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.598414    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.599350 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.598492    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhw5p\" (UniqueName: \"kubernetes.io/projected/25a060f3-9db2-4020-bacf-6b04c0941a3a-kube-api-access-xhw5p\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.599350 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.598547    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.727049 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.725589    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhw5p\" (UniqueName: \"kubernetes.io/projected/25a060f3-9db2-4020-bacf-6b04c0941a3a-kube-api-access-xhw5p\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.727049 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.725671    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.727049 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.725812    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.727049 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.725864    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.727049 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.725908    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.727049 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.725940    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a060f3-9db2-4020-bacf-6b04c0941a3a-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.727049 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.725985    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.727049 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.726055    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.727049 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.726105    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.727049 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.726746    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.733206 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.728566    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a060f3-9db2-4020-bacf-6b04c0941a3a-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.733206 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.728903    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.733206 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.729145    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.733206 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.732639    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.733206 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.732766    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.733206 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.733143    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.780898 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.780368    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhw5p\" (UniqueName: \"kubernetes.io/projected/25a060f3-9db2-4020-bacf-6b04c0941a3a-kube-api-access-xhw5p\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.786896 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.786837    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:25.796938 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.796909    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"]
Nov 03 02:31:25.930216 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:25.930179    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:26.358867 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:26.358832    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"]
Nov 03 02:31:26.366015 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:31:26.365990    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a060f3_9db2_4020_bacf_6b04c0941a3a.slice/crio-086ec5bdea47fb2cc7b83c7b2ce73383be259b2df3f517f5dd165216c79f44eb WatchSource:0}: Error finding container 086ec5bdea47fb2cc7b83c7b2ce73383be259b2df3f517f5dd165216c79f44eb: Status 404 returned error can't find the container with id 086ec5bdea47fb2cc7b83c7b2ce73383be259b2df3f517f5dd165216c79f44eb
Nov 03 02:31:26.368758 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:26.368723    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:31:26.368870 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:26.368807    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:31:26.368870 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:26.368854    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:31:26.462148 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:26.462068    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw" event={"ID":"25a060f3-9db2-4020-bacf-6b04c0941a3a","Type":"ContainerStarted","Data":"086ec5bdea47fb2cc7b83c7b2ce73383be259b2df3f517f5dd165216c79f44eb"}
Nov 03 02:31:26.489232 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:26.487342    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 02:31:26.489232 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:26.487390    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr" podUID="ecdc91da-4a69-4e9a-b46c-12d9cf909aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 02:31:27.471732 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:27.471675    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw" event={"ID":"25a060f3-9db2-4020-bacf-6b04c0941a3a","Type":"ContainerStarted","Data":"5ee308942fda5de40f54dc3a653781baf06fdb39225258311a9c5c609e66f259"}
Nov 03 02:31:27.487077 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:27.487040    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 02:31:27.487219 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:27.487101    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr" podUID="ecdc91da-4a69-4e9a-b46c-12d9cf909aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 02:31:27.674219 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:27.674149    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw" podStartSLOduration=2.674134216 podStartE2EDuration="2.674134216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:31:27.63008962 +0000 UTC m=+2414.644330986" watchObservedRunningTime="2026-11-03 02:31:27.674134216 +0000 UTC m=+2414.688375583"
Nov 03 02:31:27.940719 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:27.937587    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:27.940719 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:27.937771    5005 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 02:31:27.940719 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:27.937832    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw" podUID="25a060f3-9db2-4020-bacf-6b04c0941a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 02:31:28.487209 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:28.487174    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 02:31:28.487403 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:28.487224    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr" podUID="ecdc91da-4a69-4e9a-b46c-12d9cf909aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 02:31:28.930859 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:28.930825    5005 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 02:31:28.931321 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:28.930879    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw" podUID="25a060f3-9db2-4020-bacf-6b04c0941a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 02:31:29.370047 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:29.370000    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"]
Nov 03 02:31:29.370270 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:29.370212    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc" podUID="c99ef35d-16a1-435b-afa3-203d04df4f1c" containerName="maas-api" containerID="cri-o://b8f8701d5c0e3d005d724ccece3f6c1b33a08489dd167cc7e410a56d119f079d" gracePeriod=30
Nov 03 02:31:29.492779 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:29.492742    5005 generic.go:356] "Generic (PLEG): container finished" podID="c99ef35d-16a1-435b-afa3-203d04df4f1c" containerID="b8f8701d5c0e3d005d724ccece3f6c1b33a08489dd167cc7e410a56d119f079d" exitCode=0
Nov 03 02:31:29.492937 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:29.492816    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc" event={"ID":"c99ef35d-16a1-435b-afa3-203d04df4f1c","Type":"ContainerDied","Data":"b8f8701d5c0e3d005d724ccece3f6c1b33a08489dd167cc7e410a56d119f079d"}
Nov 03 02:31:29.872719 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:29.868466    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"
Nov 03 02:31:29.933289 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:29.933189    5005 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 02:31:29.933893 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:29.933851    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw" podUID="25a060f3-9db2-4020-bacf-6b04c0941a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 02:31:29.956639 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:29.956600    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c99ef35d-16a1-435b-afa3-203d04df4f1c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99ef35d-16a1-435b-afa3-203d04df4f1c-metrics-certs\") pod \"c99ef35d-16a1-435b-afa3-203d04df4f1c\" (UID: \"c99ef35d-16a1-435b-afa3-203d04df4f1c\") "
Nov 03 02:31:29.956850 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:29.956737    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c99ef35d-16a1-435b-afa3-203d04df4f1c-kube-api-access-7px79\" (UniqueName: \"kubernetes.io/projected/c99ef35d-16a1-435b-afa3-203d04df4f1c-kube-api-access-7px79\") pod \"c99ef35d-16a1-435b-afa3-203d04df4f1c\" (UID: \"c99ef35d-16a1-435b-afa3-203d04df4f1c\") "
Nov 03 02:31:29.956928 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:29.956870    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c99ef35d-16a1-435b-afa3-203d04df4f1c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c99ef35d-16a1-435b-afa3-203d04df4f1c-maas-api-tls\") pod \"c99ef35d-16a1-435b-afa3-203d04df4f1c\" (UID: \"c99ef35d-16a1-435b-afa3-203d04df4f1c\") "
Nov 03 02:31:29.957032 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:29.956989    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c99ef35d-16a1-435b-afa3-203d04df4f1c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c99ef35d-16a1-435b-afa3-203d04df4f1c-openshift-service-ca\") pod \"c99ef35d-16a1-435b-afa3-203d04df4f1c\" (UID: \"c99ef35d-16a1-435b-afa3-203d04df4f1c\") "
Nov 03 02:31:29.957837 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:29.957812    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c99ef35d-16a1-435b-afa3-203d04df4f1c-openshift-service-ca" pod "c99ef35d-16a1-435b-afa3-203d04df4f1c" (UID: "c99ef35d-16a1-435b-afa3-203d04df4f1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:31:29.959298 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:29.959272    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c99ef35d-16a1-435b-afa3-203d04df4f1c-kube-api-access-7px79" pod "c99ef35d-16a1-435b-afa3-203d04df4f1c" (UID: "c99ef35d-16a1-435b-afa3-203d04df4f1c"). InnerVolumeSpecName "kube-api-access-7px79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:31:29.959429 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:29.959284    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c99ef35d-16a1-435b-afa3-203d04df4f1c-metrics-certs" pod "c99ef35d-16a1-435b-afa3-203d04df4f1c" (UID: "c99ef35d-16a1-435b-afa3-203d04df4f1c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:31:29.960093 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:29.960071    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c99ef35d-16a1-435b-afa3-203d04df4f1c-maas-api-tls" pod "c99ef35d-16a1-435b-afa3-203d04df4f1c" (UID: "c99ef35d-16a1-435b-afa3-203d04df4f1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:31:30.064334 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:30.064297    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c99ef35d-16a1-435b-afa3-203d04df4f1c-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:30.064334 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:30.064337    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c99ef35d-16a1-435b-afa3-203d04df4f1c-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:30.064532 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:30.064353    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c99ef35d-16a1-435b-afa3-203d04df4f1c-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:30.064532 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:30.064370    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7px79\" (UniqueName: \"kubernetes.io/projected/c99ef35d-16a1-435b-afa3-203d04df4f1c-kube-api-access-7px79\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:30.142328 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:30.142286    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:30.144135 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:30.144081    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:30.145648 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:30.145625    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:31:30.533814 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:30.533784    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"
Nov 03 02:31:30.534276 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:30.533796    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc" event={"ID":"c99ef35d-16a1-435b-afa3-203d04df4f1c","Type":"ContainerDied","Data":"0b4e4968a2aad4d52b00ef590526cb6f694eae008615bede1b24c848ece5fcc2"}
Nov 03 02:31:30.535817 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:30.535796    5005 scope.go:122] "RemoveContainer" containerID="b8f8701d5c0e3d005d724ccece3f6c1b33a08489dd167cc7e410a56d119f079d"
Nov 03 02:31:30.765760 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:30.751788    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"]
Nov 03 02:31:30.777717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:30.774779    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8857cb-c98bb85b7-7ngvc"]
Nov 03 02:31:30.849631 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:31:30.849551    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:31:30.930630 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:30.930592    5005 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 02:31:30.930851 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:30.930673    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw" podUID="25a060f3-9db2-4020-bacf-6b04c0941a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 02:31:31.133227 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:31.133141    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c99ef35d-16a1-435b-afa3-203d04df4f1c" path="/var/lib/kubelet/pods/c99ef35d-16a1-435b-afa3-203d04df4f1c/volumes"
Nov 03 02:31:31.930782 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:31.930739    5005 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 02:31:31.931112 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:31.930803    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw" podUID="25a060f3-9db2-4020-bacf-6b04c0941a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 02:31:32.935396 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:32.935362    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:32.935854 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:32.935740    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:32.936587 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:32.936566    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:31:40.685518 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.685476    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"]
Nov 03 02:31:40.688796 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.688768    5005 cpu_manager.go:414] "containerMap: removing container" podUID="c99ef35d-16a1-435b-afa3-203d04df4f1c" containerName="maas-api"
Nov 03 02:31:40.688984 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.688970    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c99ef35d-16a1-435b-afa3-203d04df4f1c" containerName="maas-api"
Nov 03 02:31:40.689344 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.689326    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="c99ef35d-16a1-435b-afa3-203d04df4f1c" containerName="maas-api"
Nov 03 02:31:40.694379 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.694355    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.744840 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.744801    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"]
Nov 03 02:31:40.756533 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.756498    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.756712 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.756545    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20db3162-f8b1-44e3-84ad-f149146b0c1a-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.756712 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.756647    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.756818 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.756687    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.756929 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.756900    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbktj\" (UniqueName: \"kubernetes.io/projected/20db3162-f8b1-44e3-84ad-f149146b0c1a-kube-api-access-bbktj\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.756972 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.756959    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.757011 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.756996    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.757062 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.757050    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.757111 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.757091    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.858185 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.858148    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.858395 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.858215    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.858395 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.858267    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.858395 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.858300    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20db3162-f8b1-44e3-84ad-f149146b0c1a-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.858395 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.858348    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.858395 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.858391    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.858651 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.858463    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbktj\" (UniqueName: \"kubernetes.io/projected/20db3162-f8b1-44e3-84ad-f149146b0c1a-kube-api-access-bbktj\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.858651 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.858497    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.858651 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.858525    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.858826 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.858730    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-data\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.859164 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.859123    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-credential-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.859312 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.859206    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-workload-certs\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.859312 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.859306    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20db3162-f8b1-44e3-84ad-f149146b0c1a-istiod-ca-cert\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.859443 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.859353    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-workload-socket\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.861069 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.861048    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-envoy\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.861069 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.861056    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-podinfo\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.911671 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.911636    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-token\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:40.911871 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:40.911821    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbktj\" (UniqueName: \"kubernetes.io/projected/20db3162-f8b1-44e3-84ad-f149146b0c1a-kube-api-access-bbktj\") pod \"e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") " pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:41.010548 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:41.010510    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:41.441738 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:41.441635    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"]
Nov 03 02:31:41.446973 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:31:41.446941    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20db3162_f8b1_44e3_84ad_f149146b0c1a.slice/crio-a42189c075ecd8b2936953f66cbdf3ef35389a86f1154d33f84ba01173200705 WatchSource:0}: Error finding container a42189c075ecd8b2936953f66cbdf3ef35389a86f1154d33f84ba01173200705: Status 404 returned error can't find the container with id a42189c075ecd8b2936953f66cbdf3ef35389a86f1154d33f84ba01173200705
Nov 03 02:31:41.449607 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:41.449569    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:31:41.449712 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:41.449646    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:31:41.449712 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:41.449679    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:31:41.618989 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:41.618952    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b" event={"ID":"20db3162-f8b1-44e3-84ad-f149146b0c1a","Type":"ContainerStarted","Data":"d643465b087ddaf852368fe9d87dda5934e16a8e9b9e8fc333a0daba254e61e4"}
Nov 03 02:31:41.619250 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:41.619220    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b" event={"ID":"20db3162-f8b1-44e3-84ad-f149146b0c1a","Type":"ContainerStarted","Data":"a42189c075ecd8b2936953f66cbdf3ef35389a86f1154d33f84ba01173200705"}
Nov 03 02:31:41.734170 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:41.734059    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b" podStartSLOduration=1.734040399 podStartE2EDuration="1.734040399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:31:41.729213769 +0000 UTC m=+2428.743455170" watchObservedRunningTime="2026-11-03 02:31:41.734040399 +0000 UTC m=+2428.748281766"
Nov 03 02:31:42.012033 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:42.011998    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:42.013776 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:42.013746    5005 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 02:31:42.013906 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:42.013801    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b" podUID="20db3162-f8b1-44e3-84ad-f149146b0c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 02:31:43.021721 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:43.018128    5005 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 02:31:43.021721 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:43.018205    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b" podUID="20db3162-f8b1-44e3-84ad-f149146b0c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 02:31:44.017133 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:44.017090    5005 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 02:31:44.017435 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:44.017406    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b" podUID="20db3162-f8b1-44e3-84ad-f149146b0c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 02:31:45.016725 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:45.011044    5005 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 02:31:45.016725 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:45.011105    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b" podUID="20db3162-f8b1-44e3-84ad-f149146b0c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 02:31:46.020718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:46.018823    5005 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 02:31:46.020718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:46.018892    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b" podUID="20db3162-f8b1-44e3-84ad-f149146b0c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 02:31:47.027720 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:47.023096    5005 patch_prober.go:29] interesting pod/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 02:31:47.027720 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:47.023161    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b" podUID="20db3162-f8b1-44e3-84ad-f149146b0c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 02:31:48.305295 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:48.305258    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:48.307836 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:48.307536    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:48.311829 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:48.311799    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:31:51.501029 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:51.500987    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"]
Nov 03 02:31:51.501470 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:51.501239    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv" podUID="10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f" containerName="maas-api" containerID="cri-o://09cddefe9d374538ec8956d820e87418498fc1c0f6844498b9571c43102029a7" gracePeriod=30
Nov 03 02:31:51.725767 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:51.720898    5005 generic.go:356] "Generic (PLEG): container finished" podID="10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f" containerID="09cddefe9d374538ec8956d820e87418498fc1c0f6844498b9571c43102029a7" exitCode=0
Nov 03 02:31:51.725767 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:51.721132    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv" event={"ID":"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f","Type":"ContainerDied","Data":"09cddefe9d374538ec8956d820e87418498fc1c0f6844498b9571c43102029a7"}
Nov 03 02:31:51.816681 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:51.816642    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"
Nov 03 02:31:51.959383 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:51.959345    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-metrics-certs\") pod \"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f\" (UID: \"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f\") "
Nov 03 02:31:51.959586 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:51.959439    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-openshift-service-ca\") pod \"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f\" (UID: \"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f\") "
Nov 03 02:31:51.959586 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:51.959542    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-maas-api-tls\") pod \"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f\" (UID: \"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f\") "
Nov 03 02:31:51.959739 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:51.959682    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-kube-api-access-f5rhj\" (UniqueName: \"kubernetes.io/projected/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-kube-api-access-f5rhj\") pod \"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f\" (UID: \"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f\") "
Nov 03 02:31:51.959843 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:51.959818    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-openshift-service-ca" pod "10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f" (UID: "10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:31:51.960340 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:51.960313    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:51.961828 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:51.961783    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-metrics-certs" pod "10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f" (UID: "10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:31:51.962013 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:51.961990    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-maas-api-tls" pod "10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f" (UID: "10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:31:51.962296 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:51.962275    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-kube-api-access-f5rhj" pod "10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f" (UID: "10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f"). InnerVolumeSpecName "kube-api-access-f5rhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:31:52.063775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:52.062792    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:52.063775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:52.062829    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:52.063775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:52.062845    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5rhj\" (UniqueName: \"kubernetes.io/projected/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f-kube-api-access-f5rhj\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:52.730214 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:52.730175    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv" event={"ID":"10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f","Type":"ContainerDied","Data":"89c06977b3d86dc700dc46710071493415fe691a132fae4ee279534313158012"}
Nov 03 02:31:52.730214 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:52.730208    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"
Nov 03 02:31:52.730793 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:52.730246    5005 scope.go:122] "RemoveContainer" containerID="09cddefe9d374538ec8956d820e87418498fc1c0f6844498b9571c43102029a7"
Nov 03 02:31:52.884918 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:52.884883    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"]
Nov 03 02:31:52.931133 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:52.931101    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7f760c-ff9d5b76d-wp4nv"]
Nov 03 02:31:53.109409 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:31:53.109372    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f" path="/var/lib/kubelet/pods/10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f/volumes"
Nov 03 02:32:00.714386 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:32:00.714345    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:32:06.197111 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.197067    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"]
Nov 03 02:32:06.197961 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.197940    5005 cpu_manager.go:414] "containerMap: removing container" podUID="10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f" containerName="maas-api"
Nov 03 02:32:06.197961 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.197962    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f" containerName="maas-api"
Nov 03 02:32:06.198190 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.198122    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="10cbe959-14ee-4e1c-b5ae-5e3bf92beb2f" containerName="maas-api"
Nov 03 02:32:06.203865 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.203840    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.317159 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.317124    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"]
Nov 03 02:32:06.343897 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.343861    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.343897 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.343899    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xjvs\" (UniqueName: \"kubernetes.io/projected/1fc5f95b-3ced-444a-aab9-39c666ac1a29-kube-api-access-7xjvs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.344177 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.343931    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.344177 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.344118    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.344300 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.344283    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.344375 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.344313    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.344452 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.344433    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.344511 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.344480    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.344613 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.344597    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.446225 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.446186    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.446225 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.446227    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xjvs\" (UniqueName: \"kubernetes.io/projected/1fc5f95b-3ced-444a-aab9-39c666ac1a29-kube-api-access-7xjvs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.446473 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.446260    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.446473 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.446324    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.446473 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.446373    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.446473 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.446399    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.446473 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.446459    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.446751 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.446516    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.446751 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.446607    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.446865 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.446828    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-data\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.446922 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.446869    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-workload-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.446974 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.446958    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-workload-certs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.447021 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.446976    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-credential-socket\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.447517 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.447459    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istiod-ca-cert\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.448584 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.448562    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-envoy\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.449306 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.449286    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-podinfo\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.500216 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.500186    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xjvs\" (UniqueName: \"kubernetes.io/projected/1fc5f95b-3ced-444a-aab9-39c666ac1a29-kube-api-access-7xjvs\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.500376 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.500326    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-token\") pod \"e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") " pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.548636 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.548599    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:06.979110 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.979070    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"]
Nov 03 02:32:06.985001 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:32:06.984970    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc5f95b_3ced_444a_aab9_39c666ac1a29.slice/crio-d77bb10d12f35284cc3dfdcb0cc143c202b888b9e08ff24df4afd49a78de3ebc WatchSource:0}: Error finding container d77bb10d12f35284cc3dfdcb0cc143c202b888b9e08ff24df4afd49a78de3ebc: Status 404 returned error can't find the container with id d77bb10d12f35284cc3dfdcb0cc143c202b888b9e08ff24df4afd49a78de3ebc
Nov 03 02:32:06.987365 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.987325    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:32:06.987474 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.987416    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:32:06.987474 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:06.987463    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:32:07.902745 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:07.902687    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr" event={"ID":"1fc5f95b-3ced-444a-aab9-39c666ac1a29","Type":"ContainerStarted","Data":"89e3c0b1ac2138e9f48f3aa6e80a49669499420a7cb6592661da37aaf1cdbc1f"}
Nov 03 02:32:07.903204 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:07.902779    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr" event={"ID":"1fc5f95b-3ced-444a-aab9-39c666ac1a29","Type":"ContainerStarted","Data":"d77bb10d12f35284cc3dfdcb0cc143c202b888b9e08ff24df4afd49a78de3ebc"}
Nov 03 02:32:08.061599 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:08.061541    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr" podStartSLOduration=2.061525822 podStartE2EDuration="2.061525822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:32:08.060875805 +0000 UTC m=+2455.075117387" watchObservedRunningTime="2026-11-03 02:32:08.061525822 +0000 UTC m=+2455.075767188"
Nov 03 02:32:08.557718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:08.553712    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:08.557718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:08.556750    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 02:32:08.557718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:08.556827    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr" podUID="1fc5f95b-3ced-444a-aab9-39c666ac1a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 02:32:09.553506 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:09.553468    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 02:32:09.553925 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:09.553526    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr" podUID="1fc5f95b-3ced-444a-aab9-39c666ac1a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 02:32:10.553933 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:10.553893    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 02:32:10.554348 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:10.553966    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr" podUID="1fc5f95b-3ced-444a-aab9-39c666ac1a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 02:32:11.554351 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.554311    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 02:32:11.554798 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.554380    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr" podUID="1fc5f95b-3ced-444a-aab9-39c666ac1a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 02:32:11.568937 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.568895    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"]
Nov 03 02:32:11.573320 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.573299    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.629223 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.629186    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-1bccbd-openshift-default-dockercfg-c57w7\""
Nov 03 02:32:11.640961 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.640927    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.641118 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.641060    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b97ff37-eae3-4020-86ed-71477662e225-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.641118 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.641092    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.641194 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.641122    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b97ff37-eae3-4020-86ed-71477662e225-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.641194 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.641139    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.641329 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.641305    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqr8q\" (UniqueName: \"kubernetes.io/projected/5b97ff37-eae3-4020-86ed-71477662e225-kube-api-access-bqr8q\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.641466 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.641423    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b97ff37-eae3-4020-86ed-71477662e225-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.641519 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.641503    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.641565 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.641550    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.677326 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.677285    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"]
Nov 03 02:32:11.743393 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.743359    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.743584 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.743405    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.743584 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.743525    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.743720 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.743674    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b97ff37-eae3-4020-86ed-71477662e225-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.743790 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.743771    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.743857 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.743815    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.743857 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.743821    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b97ff37-eae3-4020-86ed-71477662e225-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.743857 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.743853    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.743999 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.743855    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.743999 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.743894    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.743999 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.743931    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqr8q\" (UniqueName: \"kubernetes.io/projected/5b97ff37-eae3-4020-86ed-71477662e225-kube-api-access-bqr8q\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.744146 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.744023    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b97ff37-eae3-4020-86ed-71477662e225-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.744146 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.744078    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.744724 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.744669    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b97ff37-eae3-4020-86ed-71477662e225-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.746120 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.746099    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.746212 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.746194    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b97ff37-eae3-4020-86ed-71477662e225-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.797471 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.797440    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqr8q\" (UniqueName: \"kubernetes.io/projected/5b97ff37-eae3-4020-86ed-71477662e225-kube-api-access-bqr8q\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.797621 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.797485    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b97ff37-eae3-4020-86ed-71477662e225-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:11.887524 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:11.887439    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:12.346398 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:12.346363    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"]
Nov 03 02:32:12.352036 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:32:12.352004    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b97ff37_eae3_4020_86ed_71477662e225.slice/crio-326e15003073dabdbbe77df276741d99280e35079336707a4789e88e7e0e9cf1 WatchSource:0}: Error finding container 326e15003073dabdbbe77df276741d99280e35079336707a4789e88e7e0e9cf1: Status 404 returned error can't find the container with id 326e15003073dabdbbe77df276741d99280e35079336707a4789e88e7e0e9cf1
Nov 03 02:32:12.354431 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:12.354396    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:32:12.354522 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:12.354468    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:32:12.354522 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:12.354497    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:32:12.553552 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:12.553512    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 02:32:12.553749 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:12.553572    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr" podUID="1fc5f95b-3ced-444a-aab9-39c666ac1a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 02:32:12.957223 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:12.957186    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h" event={"ID":"5b97ff37-eae3-4020-86ed-71477662e225","Type":"ContainerStarted","Data":"85653861162e40bd77747f682139ef697c7c547a0cc35ae1ac652629c9b08633"}
Nov 03 02:32:12.957656 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:12.957247    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h" event={"ID":"5b97ff37-eae3-4020-86ed-71477662e225","Type":"ContainerStarted","Data":"326e15003073dabdbbe77df276741d99280e35079336707a4789e88e7e0e9cf1"}
Nov 03 02:32:13.073372 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:13.073319    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h" podStartSLOduration=2.073304762 podStartE2EDuration="2.073304762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:32:13.071230985 +0000 UTC m=+2460.085472406" watchObservedRunningTime="2026-11-03 02:32:13.073304762 +0000 UTC m=+2460.087546127"
Nov 03 02:32:13.554221 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:13.554187    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 02:32:13.554396 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:13.554240    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr" podUID="1fc5f95b-3ced-444a-aab9-39c666ac1a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 02:32:13.902320 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:13.897013    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:13.924718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:13.923924    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 02:32:13.924718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:13.924015    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h" podUID="5b97ff37-eae3-4020-86ed-71477662e225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 02:32:14.553686 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:14.553642    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 02:32:14.554114 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:14.553738    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr" podUID="1fc5f95b-3ced-444a-aab9-39c666ac1a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 02:32:14.899789 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:14.891163    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 02:32:14.899789 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:14.891229    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h" podUID="5b97ff37-eae3-4020-86ed-71477662e225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 02:32:15.888286 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:15.888241    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 02:32:15.888841 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:15.888301    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h" podUID="5b97ff37-eae3-4020-86ed-71477662e225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 02:32:16.601855 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:16.601812    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:32:16.602172 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:16.602131    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr" podUID="1fc5f95b-3ced-444a-aab9-39c666ac1a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:32:16.888543 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:16.888453    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 02:32:16.888543 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:16.888510    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h" podUID="5b97ff37-eae3-4020-86ed-71477662e225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 02:32:16.967124 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:16.967084    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:16.967416 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:16.967393    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:16.971932 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:16.971903    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:32:17.888484 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:17.888441    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 02:32:17.888753 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:17.888517    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h" podUID="5b97ff37-eae3-4020-86ed-71477662e225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 02:32:18.888778 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:18.888732    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 02:32:18.889216 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:18.888807    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h" podUID="5b97ff37-eae3-4020-86ed-71477662e225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 02:32:20.929929 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:20.929884    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:32:20.930460 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:20.929955    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h" podUID="5b97ff37-eae3-4020-86ed-71477662e225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:32:21.235131 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:21.235044    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:21.235995 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:21.235968    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:21.236655 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:21.236631    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:32:30.717956 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:32:30.717861    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:32:44.550476 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:44.550444    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-dn4d4_30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2/payload-processing/0.log"
Nov 03 02:32:44.778428 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:44.778396    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-dn4d4_30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2/payload-processing/0.log"
Nov 03 02:32:45.807990 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:45.807956    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"]
Nov 03 02:32:45.816726 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:45.816667    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6" podUID="2114664b-4364-4e6e-a1bc-cf54d2c4d8ce" containerName="maas-api" containerID="cri-o://33ca0d811633c83b705725c63b9f94c75967490eb80a9b7eac6271557ac638b4" gracePeriod=30
Nov 03 02:32:46.140041 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:46.140016    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"
Nov 03 02:32:46.220557 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:46.220515    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-openshift-service-ca\") pod \"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce\" (UID: \"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce\") "
Nov 03 02:32:46.220799 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:46.220606    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-maas-api-tls\") pod \"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce\" (UID: \"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce\") "
Nov 03 02:32:46.220878 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:46.220834    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-kube-api-access-xbz78\" (UniqueName: \"kubernetes.io/projected/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-kube-api-access-xbz78\") pod \"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce\" (UID: \"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce\") "
Nov 03 02:32:46.220994 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:46.220975    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-metrics-certs\") pod \"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce\" (UID: \"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce\") "
Nov 03 02:32:46.222292 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:46.222252    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-openshift-service-ca" pod "2114664b-4364-4e6e-a1bc-cf54d2c4d8ce" (UID: "2114664b-4364-4e6e-a1bc-cf54d2c4d8ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:32:46.240000 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:46.239963    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-maas-api-tls" pod "2114664b-4364-4e6e-a1bc-cf54d2c4d8ce" (UID: "2114664b-4364-4e6e-a1bc-cf54d2c4d8ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:32:46.242503 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:46.242411    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-metrics-certs" pod "2114664b-4364-4e6e-a1bc-cf54d2c4d8ce" (UID: "2114664b-4364-4e6e-a1bc-cf54d2c4d8ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:32:46.242866 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:46.242839    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-kube-api-access-xbz78" pod "2114664b-4364-4e6e-a1bc-cf54d2c4d8ce" (UID: "2114664b-4364-4e6e-a1bc-cf54d2c4d8ce"). InnerVolumeSpecName "kube-api-access-xbz78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:32:46.323032 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:46.322998    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbz78\" (UniqueName: \"kubernetes.io/projected/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-kube-api-access-xbz78\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:32:46.323032 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:46.323030    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:32:46.323032 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:46.323040    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:32:46.323265 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:46.323049    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:32:46.861043 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:46.861001    5005 generic.go:356] "Generic (PLEG): container finished" podID="2114664b-4364-4e6e-a1bc-cf54d2c4d8ce" containerID="33ca0d811633c83b705725c63b9f94c75967490eb80a9b7eac6271557ac638b4" exitCode=0
Nov 03 02:32:46.861552 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:46.861071    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"
Nov 03 02:32:46.861552 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:46.861092    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6" event={"ID":"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce","Type":"ContainerDied","Data":"33ca0d811633c83b705725c63b9f94c75967490eb80a9b7eac6271557ac638b4"}
Nov 03 02:32:46.861552 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:46.861165    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6" event={"ID":"2114664b-4364-4e6e-a1bc-cf54d2c4d8ce","Type":"ContainerDied","Data":"227d619d8fb4de888514d2c7209f102a7f336c2deb7b86039faea3ef06523912"}
Nov 03 02:32:46.861552 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:46.861195    5005 scope.go:122] "RemoveContainer" containerID="33ca0d811633c83b705725c63b9f94c75967490eb80a9b7eac6271557ac638b4"
Nov 03 02:32:46.874649 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:46.874621    5005 scope.go:122] "RemoveContainer" containerID="33ca0d811633c83b705725c63b9f94c75967490eb80a9b7eac6271557ac638b4"
Nov 03 02:32:46.875010 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:32:46.874990    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33ca0d811633c83b705725c63b9f94c75967490eb80a9b7eac6271557ac638b4\": container with ID starting with 33ca0d811633c83b705725c63b9f94c75967490eb80a9b7eac6271557ac638b4 not found: ID does not exist" containerID="33ca0d811633c83b705725c63b9f94c75967490eb80a9b7eac6271557ac638b4"
Nov 03 02:32:46.875096 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:46.875018    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33ca0d811633c83b705725c63b9f94c75967490eb80a9b7eac6271557ac638b4"} err="failed to get container status \"33ca0d811633c83b705725c63b9f94c75967490eb80a9b7eac6271557ac638b4\": rpc error: code = NotFound desc = could not find container \"33ca0d811633c83b705725c63b9f94c75967490eb80a9b7eac6271557ac638b4\": container with ID starting with 33ca0d811633c83b705725c63b9f94c75967490eb80a9b7eac6271557ac638b4 not found: ID does not exist"
Nov 03 02:32:47.016030 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:47.015993    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"]
Nov 03 02:32:47.062312 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:47.062269    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-711318-cb98bcb67-dv2k6"]
Nov 03 02:32:47.110810 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:32:47.110776    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2114664b-4364-4e6e-a1bc-cf54d2c4d8ce" path="/var/lib/kubelet/pods/2114664b-4364-4e6e-a1bc-cf54d2c4d8ce/volumes"
Nov 03 02:33:00.714385 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:33:00.714342    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:33:27.114444 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.114411    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"]
Nov 03 02:33:27.115138 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.115049    5005 cpu_manager.go:414] "containerMap: removing container" podUID="2114664b-4364-4e6e-a1bc-cf54d2c4d8ce" containerName="maas-api"
Nov 03 02:33:27.115138 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.115064    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2114664b-4364-4e6e-a1bc-cf54d2c4d8ce" containerName="maas-api"
Nov 03 02:33:27.115227 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.115163    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="2114664b-4364-4e6e-a1bc-cf54d2c4d8ce" containerName="maas-api"
Nov 03 02:33:27.119844 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.119824    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.195179 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.195142    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.195386 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.195287    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.195386 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.195368    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.195524 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.195400    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.195524 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.195467    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.195524 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.195487    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.195524 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.195519    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.195816 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.195543    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.195816 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.195670    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghn27\" (UniqueName: \"kubernetes.io/projected/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-kube-api-access-ghn27\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.218349 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.218317    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"]
Nov 03 02:33:27.296707 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.296653    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.296904 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.296746    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.296904 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.296811    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.296904 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.296838    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.296904 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.296877    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.297147 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.296916    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.297147 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.296991    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghn27\" (UniqueName: \"kubernetes.io/projected/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-kube-api-access-ghn27\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.297147 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.297086    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.297313 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.297145    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-data\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.297313 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.297185    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-workload-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.297313 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.297209    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.297476 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.297379    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-credential-socket\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.297855 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.297830    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-workload-certs\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.298111 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.298080    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istiod-ca-cert\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.299583 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.299557    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-envoy\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.300087 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.300066    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-podinfo\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.350892 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.350853    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-token\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.351089 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.351065    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghn27\" (UniqueName: \"kubernetes.io/projected/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-kube-api-access-ghn27\") pod \"e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") " pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.436408 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.436318    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:27.901815 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.901783    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"]
Nov 03 02:33:27.907555 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:33:27.907521    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa108bd_7651_4fe9_9e46_3d9ef40139fe.slice/crio-b238b42ec7414636ecdc1ca93b7e6c8c07fd58d1cf3e1977d79bfcd18b4876b1 WatchSource:0}: Error finding container b238b42ec7414636ecdc1ca93b7e6c8c07fd58d1cf3e1977d79bfcd18b4876b1: Status 404 returned error can't find the container with id b238b42ec7414636ecdc1ca93b7e6c8c07fd58d1cf3e1977d79bfcd18b4876b1
Nov 03 02:33:27.909822 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.909784    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:33:27.909914 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.909858    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:33:27.909914 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:27.909887    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:33:28.390861 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:28.390820    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4" event={"ID":"bfa108bd-7651-4fe9-9e46-3d9ef40139fe","Type":"ContainerStarted","Data":"5d5081af51cba7933c980912d182143a0561b8402411cf24282dcafdbb16f915"}
Nov 03 02:33:28.391341 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:28.390877    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4" event={"ID":"bfa108bd-7651-4fe9-9e46-3d9ef40139fe","Type":"ContainerStarted","Data":"b238b42ec7414636ecdc1ca93b7e6c8c07fd58d1cf3e1977d79bfcd18b4876b1"}
Nov 03 02:33:28.437506 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:28.437462    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:28.439456 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:28.439427    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 02:33:28.439588 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:28.439481    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4" podUID="bfa108bd-7651-4fe9-9e46-3d9ef40139fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 02:33:28.558262 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:28.558204    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4" podStartSLOduration=1.5581897630000001 podStartE2EDuration="1.558189763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:28.509195887 +0000 UTC m=+2535.523437293" watchObservedRunningTime="2026-11-03 02:33:28.558189763 +0000 UTC m=+2535.572431128"
Nov 03 02:33:29.444730 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:29.444286    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 02:33:29.444730 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:29.444348    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4" podUID="bfa108bd-7651-4fe9-9e46-3d9ef40139fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 02:33:30.437718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:30.437236    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 02:33:30.437718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:30.437295    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4" podUID="bfa108bd-7651-4fe9-9e46-3d9ef40139fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 02:33:30.917720 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:33:30.917585    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:33:31.437343 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:31.437308    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 02:33:31.437542 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:31.437375    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4" podUID="bfa108bd-7651-4fe9-9e46-3d9ef40139fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 02:33:32.437608 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:32.437572    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 02:33:32.438023 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:32.437636    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4" podUID="bfa108bd-7651-4fe9-9e46-3d9ef40139fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 02:33:33.441211 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:33.441170    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 02:33:33.443509 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:33.443473    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4" podUID="bfa108bd-7651-4fe9-9e46-3d9ef40139fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 02:33:34.437123 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:34.437083    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 03 02:33:34.437319 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:34.437141    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4" podUID="bfa108bd-7651-4fe9-9e46-3d9ef40139fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 03 02:33:36.477887 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:36.477846    5005 patch_prober.go:29] interesting pod/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:33:36.478404 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:36.477918    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4" podUID="bfa108bd-7651-4fe9-9e46-3d9ef40139fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:33:36.912880 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:36.912830    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:36.913227 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:36.913175    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:36.914162 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:36.914142    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:33:37.214653 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.214570    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"]
Nov 03 02:33:37.214864 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.214826    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j" podUID="ae2fd3df-1c03-4b71-8b58-0b7532fbd668" containerName="maas-api" containerID="cri-o://368816be067415171fce80b9e6ff08b47112a7ef03178e444aed34e4cdae287a" gracePeriod=30
Nov 03 02:33:37.478827 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.478762    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"
Nov 03 02:33:37.557483 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.557448    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-metrics-certs\") pod \"ae2fd3df-1c03-4b71-8b58-0b7532fbd668\" (UID: \"ae2fd3df-1c03-4b71-8b58-0b7532fbd668\") "
Nov 03 02:33:37.557646 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.557493    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-kube-api-access-925rq\" (UniqueName: \"kubernetes.io/projected/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-kube-api-access-925rq\") pod \"ae2fd3df-1c03-4b71-8b58-0b7532fbd668\" (UID: \"ae2fd3df-1c03-4b71-8b58-0b7532fbd668\") "
Nov 03 02:33:37.557646 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.557522    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-openshift-service-ca\") pod \"ae2fd3df-1c03-4b71-8b58-0b7532fbd668\" (UID: \"ae2fd3df-1c03-4b71-8b58-0b7532fbd668\") "
Nov 03 02:33:37.557646 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.557583    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-maas-api-tls\") pod \"ae2fd3df-1c03-4b71-8b58-0b7532fbd668\" (UID: \"ae2fd3df-1c03-4b71-8b58-0b7532fbd668\") "
Nov 03 02:33:37.557996 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.557967    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-openshift-service-ca" pod "ae2fd3df-1c03-4b71-8b58-0b7532fbd668" (UID: "ae2fd3df-1c03-4b71-8b58-0b7532fbd668"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:33:37.558181 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.558164    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:37.559604 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.559578    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-metrics-certs" pod "ae2fd3df-1c03-4b71-8b58-0b7532fbd668" (UID: "ae2fd3df-1c03-4b71-8b58-0b7532fbd668"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:33:37.560122 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.560097    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-maas-api-tls" pod "ae2fd3df-1c03-4b71-8b58-0b7532fbd668" (UID: "ae2fd3df-1c03-4b71-8b58-0b7532fbd668"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:33:37.560208 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.560147    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-kube-api-access-925rq" pod "ae2fd3df-1c03-4b71-8b58-0b7532fbd668" (UID: "ae2fd3df-1c03-4b71-8b58-0b7532fbd668"). InnerVolumeSpecName "kube-api-access-925rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:33:37.602139 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.602103    5005 generic.go:356] "Generic (PLEG): container finished" podID="ae2fd3df-1c03-4b71-8b58-0b7532fbd668" containerID="368816be067415171fce80b9e6ff08b47112a7ef03178e444aed34e4cdae287a" exitCode=0
Nov 03 02:33:37.602326 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.602141    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j" event={"ID":"ae2fd3df-1c03-4b71-8b58-0b7532fbd668","Type":"ContainerDied","Data":"368816be067415171fce80b9e6ff08b47112a7ef03178e444aed34e4cdae287a"}
Nov 03 02:33:37.602326 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.602187    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"
Nov 03 02:33:37.602326 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.602211    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j" event={"ID":"ae2fd3df-1c03-4b71-8b58-0b7532fbd668","Type":"ContainerDied","Data":"dc0616928a193f96e9397224fa72af5ce48fe114728ef39d00ad392c25e71f28"}
Nov 03 02:33:37.602326 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.602239    5005 scope.go:122] "RemoveContainer" containerID="368816be067415171fce80b9e6ff08b47112a7ef03178e444aed34e4cdae287a"
Nov 03 02:33:37.615427 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.615398    5005 scope.go:122] "RemoveContainer" containerID="368816be067415171fce80b9e6ff08b47112a7ef03178e444aed34e4cdae287a"
Nov 03 02:33:37.615728 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:33:37.615707    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368816be067415171fce80b9e6ff08b47112a7ef03178e444aed34e4cdae287a\": container with ID starting with 368816be067415171fce80b9e6ff08b47112a7ef03178e444aed34e4cdae287a not found: ID does not exist" containerID="368816be067415171fce80b9e6ff08b47112a7ef03178e444aed34e4cdae287a"
Nov 03 02:33:37.615794 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.615736    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368816be067415171fce80b9e6ff08b47112a7ef03178e444aed34e4cdae287a"} err="failed to get container status \"368816be067415171fce80b9e6ff08b47112a7ef03178e444aed34e4cdae287a\": rpc error: code = NotFound desc = could not find container \"368816be067415171fce80b9e6ff08b47112a7ef03178e444aed34e4cdae287a\": container with ID starting with 368816be067415171fce80b9e6ff08b47112a7ef03178e444aed34e4cdae287a not found: ID does not exist"
Nov 03 02:33:37.659418 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.659376    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:37.659418 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.659411    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-925rq\" (UniqueName: \"kubernetes.io/projected/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-kube-api-access-925rq\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:37.659624 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.659429    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae2fd3df-1c03-4b71-8b58-0b7532fbd668-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:37.760984 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.760016    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"]
Nov 03 02:33:37.772076 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:37.772044    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f9bff9-855d4b89b6-brj2j"]
Nov 03 02:33:39.109845 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:39.109812    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae2fd3df-1c03-4b71-8b58-0b7532fbd668" path="/var/lib/kubelet/pods/ae2fd3df-1c03-4b71-8b58-0b7532fbd668/volumes"
Nov 03 02:33:41.508132 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:41.508099    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"]
Nov 03 02:33:41.508616 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:41.508326    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw" podUID="25a060f3-9db2-4020-bacf-6b04c0941a3a" containerName="istio-proxy" containerID="cri-o://5ee308942fda5de40f54dc3a653781baf06fdb39225258311a9c5c609e66f259" gracePeriod=30
Nov 03 02:33:41.520768 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:41.520679    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"]
Nov 03 02:33:41.520989 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:41.520958    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b" podUID="20db3162-f8b1-44e3-84ad-f149146b0c1a" containerName="istio-proxy" containerID="cri-o://d643465b087ddaf852368fe9d87dda5934e16a8e9b9e8fc333a0daba254e61e4" gracePeriod=30
Nov 03 02:33:41.554111 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:41.554078    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"]
Nov 03 02:33:41.554288 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:41.554269    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6" podUID="05470e4a-2ba0-4978-bd4f-99040bd6119d" containerName="istio-proxy" containerID="cri-o://80db72b5ba940dc209193d3803a9f2ae5bddd69cb83366b41845752a925885c2" gracePeriod=30
Nov 03 02:33:46.837146 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.836956    5005 generic.go:356] "Generic (PLEG): container finished" podID="25a060f3-9db2-4020-bacf-6b04c0941a3a" containerID="5ee308942fda5de40f54dc3a653781baf06fdb39225258311a9c5c609e66f259" exitCode=0
Nov 03 02:33:46.837146 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.837039    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw" event={"ID":"25a060f3-9db2-4020-bacf-6b04c0941a3a","Type":"ContainerDied","Data":"5ee308942fda5de40f54dc3a653781baf06fdb39225258311a9c5c609e66f259"}
Nov 03 02:33:46.840247 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.840218    5005 generic.go:356] "Generic (PLEG): container finished" podID="20db3162-f8b1-44e3-84ad-f149146b0c1a" containerID="d643465b087ddaf852368fe9d87dda5934e16a8e9b9e8fc333a0daba254e61e4" exitCode=0
Nov 03 02:33:46.840443 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.840423    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b" event={"ID":"20db3162-f8b1-44e3-84ad-f149146b0c1a","Type":"ContainerDied","Data":"d643465b087ddaf852368fe9d87dda5934e16a8e9b9e8fc333a0daba254e61e4"}
Nov 03 02:33:46.840506 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.840473    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b" event={"ID":"20db3162-f8b1-44e3-84ad-f149146b0c1a","Type":"ContainerDied","Data":"a42189c075ecd8b2936953f66cbdf3ef35389a86f1154d33f84ba01173200705"}
Nov 03 02:33:46.840506 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.840498    5005 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a42189c075ecd8b2936953f66cbdf3ef35389a86f1154d33f84ba01173200705"
Nov 03 02:33:46.842833 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.842641    5005 generic.go:356] "Generic (PLEG): container finished" podID="05470e4a-2ba0-4978-bd4f-99040bd6119d" containerID="80db72b5ba940dc209193d3803a9f2ae5bddd69cb83366b41845752a925885c2" exitCode=0
Nov 03 02:33:46.842833 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.842739    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6" event={"ID":"05470e4a-2ba0-4978-bd4f-99040bd6119d","Type":"ContainerDied","Data":"80db72b5ba940dc209193d3803a9f2ae5bddd69cb83366b41845752a925885c2"}
Nov 03 02:33:46.849680 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.849662    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:33:46.880064 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.880034    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:33:46.923616 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.923586    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:33:46.954968 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.954912    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-envoy\") pod \"20db3162-f8b1-44e3-84ad-f149146b0c1a\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") "
Nov 03 02:33:46.955156 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.954998    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-workload-certs\") pod \"20db3162-f8b1-44e3-84ad-f149146b0c1a\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") "
Nov 03 02:33:46.955156 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.955038    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20db3162-f8b1-44e3-84ad-f149146b0c1a-kube-api-access-bbktj\" (UniqueName: \"kubernetes.io/projected/20db3162-f8b1-44e3-84ad-f149146b0c1a-kube-api-access-bbktj\") pod \"20db3162-f8b1-44e3-84ad-f149146b0c1a\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") "
Nov 03 02:33:46.955156 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.955139    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-credential-socket\") pod \"20db3162-f8b1-44e3-84ad-f149146b0c1a\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") "
Nov 03 02:33:46.955333 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.955245    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-token\" (UniqueName: \"kubernetes.io/projected/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-token\") pod \"20db3162-f8b1-44e3-84ad-f149146b0c1a\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") "
Nov 03 02:33:46.955333 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.955312    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-workload-socket\") pod \"20db3162-f8b1-44e3-84ad-f149146b0c1a\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") "
Nov 03 02:33:46.955437 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.955386    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-podinfo\") pod \"20db3162-f8b1-44e3-84ad-f149146b0c1a\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") "
Nov 03 02:33:46.955437 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.955427    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20db3162-f8b1-44e3-84ad-f149146b0c1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20db3162-f8b1-44e3-84ad-f149146b0c1a-istiod-ca-cert\") pod \"20db3162-f8b1-44e3-84ad-f149146b0c1a\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") "
Nov 03 02:33:46.955549 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.955463    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-data\") pod \"20db3162-f8b1-44e3-84ad-f149146b0c1a\" (UID: \"20db3162-f8b1-44e3-84ad-f149146b0c1a\") "
Nov 03 02:33:46.958331 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.956374    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-data" pod "20db3162-f8b1-44e3-84ad-f149146b0c1a" (UID: "20db3162-f8b1-44e3-84ad-f149146b0c1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:33:46.958331 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.956485    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-workload-certs" pod "20db3162-f8b1-44e3-84ad-f149146b0c1a" (UID: "20db3162-f8b1-44e3-84ad-f149146b0c1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:33:46.958331 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.956653    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-workload-socket" pod "20db3162-f8b1-44e3-84ad-f149146b0c1a" (UID: "20db3162-f8b1-44e3-84ad-f149146b0c1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:33:46.958331 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.957434    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20db3162-f8b1-44e3-84ad-f149146b0c1a-istiod-ca-cert" pod "20db3162-f8b1-44e3-84ad-f149146b0c1a" (UID: "20db3162-f8b1-44e3-84ad-f149146b0c1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:33:46.958331 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.957776    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-credential-socket" pod "20db3162-f8b1-44e3-84ad-f149146b0c1a" (UID: "20db3162-f8b1-44e3-84ad-f149146b0c1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:33:46.962137 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.962109    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20db3162-f8b1-44e3-84ad-f149146b0c1a-kube-api-access-bbktj" pod "20db3162-f8b1-44e3-84ad-f149146b0c1a" (UID: "20db3162-f8b1-44e3-84ad-f149146b0c1a"). InnerVolumeSpecName "kube-api-access-bbktj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:33:46.962282 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.962260    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-token" pod "20db3162-f8b1-44e3-84ad-f149146b0c1a" (UID: "20db3162-f8b1-44e3-84ad-f149146b0c1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:33:46.962436 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.962417    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-envoy" pod "20db3162-f8b1-44e3-84ad-f149146b0c1a" (UID: "20db3162-f8b1-44e3-84ad-f149146b0c1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:33:46.964197 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:46.964174    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-podinfo" pod "20db3162-f8b1-44e3-84ad-f149146b0c1a" (UID: "20db3162-f8b1-44e3-84ad-f149146b0c1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:33:47.057280 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.057235    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-data\") pod \"05470e4a-2ba0-4978-bd4f-99040bd6119d\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") "
Nov 03 02:33:47.057493 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.057323    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05470e4a-2ba0-4978-bd4f-99040bd6119d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05470e4a-2ba0-4978-bd4f-99040bd6119d-istiod-ca-cert\") pod \"05470e4a-2ba0-4978-bd4f-99040bd6119d\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") "
Nov 03 02:33:47.057493 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.057362    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25a060f3-9db2-4020-bacf-6b04c0941a3a-kube-api-access-xhw5p\" (UniqueName: \"kubernetes.io/projected/25a060f3-9db2-4020-bacf-6b04c0941a3a-kube-api-access-xhw5p\") pod \"25a060f3-9db2-4020-bacf-6b04c0941a3a\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") "
Nov 03 02:33:47.057493 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.057407    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-workload-certs\") pod \"05470e4a-2ba0-4978-bd4f-99040bd6119d\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") "
Nov 03 02:33:47.057493 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.057477    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-credential-socket\") pod \"05470e4a-2ba0-4978-bd4f-99040bd6119d\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") "
Nov 03 02:33:47.057737 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.057529    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-workload-certs\") pod \"25a060f3-9db2-4020-bacf-6b04c0941a3a\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") "
Nov 03 02:33:47.057737 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.057533    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-data" pod "05470e4a-2ba0-4978-bd4f-99040bd6119d" (UID: "05470e4a-2ba0-4978-bd4f-99040bd6119d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:33:47.057737 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.057612    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-token\" (UniqueName: \"kubernetes.io/projected/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-token\") pod \"25a060f3-9db2-4020-bacf-6b04c0941a3a\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") "
Nov 03 02:33:47.057737 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.057669    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-envoy\") pod \"05470e4a-2ba0-4978-bd4f-99040bd6119d\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") "
Nov 03 02:33:47.057951 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.057734    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25a060f3-9db2-4020-bacf-6b04c0941a3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a060f3-9db2-4020-bacf-6b04c0941a3a-istiod-ca-cert\") pod \"25a060f3-9db2-4020-bacf-6b04c0941a3a\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") "
Nov 03 02:33:47.057951 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.057743    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-workload-certs" pod "05470e4a-2ba0-4978-bd4f-99040bd6119d" (UID: "05470e4a-2ba0-4978-bd4f-99040bd6119d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:33:47.057951 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.057792    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-podinfo\") pod \"25a060f3-9db2-4020-bacf-6b04c0941a3a\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") "
Nov 03 02:33:47.057951 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.057790    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-credential-socket" pod "05470e4a-2ba0-4978-bd4f-99040bd6119d" (UID: "05470e4a-2ba0-4978-bd4f-99040bd6119d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:33:47.057951 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.057862    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-token\" (UniqueName: \"kubernetes.io/projected/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-token\") pod \"05470e4a-2ba0-4978-bd4f-99040bd6119d\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") "
Nov 03 02:33:47.057951 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.057840    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05470e4a-2ba0-4978-bd4f-99040bd6119d-istiod-ca-cert" pod "05470e4a-2ba0-4978-bd4f-99040bd6119d" (UID: "05470e4a-2ba0-4978-bd4f-99040bd6119d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:33:47.057951 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.057886    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-workload-certs" pod "25a060f3-9db2-4020-bacf-6b04c0941a3a" (UID: "25a060f3-9db2-4020-bacf-6b04c0941a3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:33:47.057951 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.057936    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-workload-socket\") pod \"25a060f3-9db2-4020-bacf-6b04c0941a3a\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") "
Nov 03 02:33:47.058385 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.057982    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-credential-socket\") pod \"25a060f3-9db2-4020-bacf-6b04c0941a3a\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") "
Nov 03 02:33:47.058385 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.058110    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25a060f3-9db2-4020-bacf-6b04c0941a3a-istiod-ca-cert" pod "25a060f3-9db2-4020-bacf-6b04c0941a3a" (UID: "25a060f3-9db2-4020-bacf-6b04c0941a3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:33:47.058385 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.058219    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-podinfo\") pod \"05470e4a-2ba0-4978-bd4f-99040bd6119d\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") "
Nov 03 02:33:47.058385 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.058228    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-workload-socket" pod "25a060f3-9db2-4020-bacf-6b04c0941a3a" (UID: "25a060f3-9db2-4020-bacf-6b04c0941a3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:33:47.058385 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.058267    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-envoy\") pod \"25a060f3-9db2-4020-bacf-6b04c0941a3a\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") "
Nov 03 02:33:47.058385 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.058296    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05470e4a-2ba0-4978-bd4f-99040bd6119d-kube-api-access-9c8nr\" (UniqueName: \"kubernetes.io/projected/05470e4a-2ba0-4978-bd4f-99040bd6119d-kube-api-access-9c8nr\") pod \"05470e4a-2ba0-4978-bd4f-99040bd6119d\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") "
Nov 03 02:33:47.058385 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.058338    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-workload-socket\") pod \"05470e4a-2ba0-4978-bd4f-99040bd6119d\" (UID: \"05470e4a-2ba0-4978-bd4f-99040bd6119d\") "
Nov 03 02:33:47.058385 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.058369    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-data\") pod \"25a060f3-9db2-4020-bacf-6b04c0941a3a\" (UID: \"25a060f3-9db2-4020-bacf-6b04c0941a3a\") "
Nov 03 02:33:47.058772 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.058414    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-credential-socket" pod "25a060f3-9db2-4020-bacf-6b04c0941a3a" (UID: "25a060f3-9db2-4020-bacf-6b04c0941a3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:33:47.058974 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.058956    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05470e4a-2ba0-4978-bd4f-99040bd6119d-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.059094 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.058978    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.059094 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.058992    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.059094 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.059006    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.059094 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.059019    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.059094 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.059032    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20db3162-f8b1-44e3-84ad-f149146b0c1a-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.059094 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.059045    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.059094 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.059058    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.059094 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.059071    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.059094 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.059085    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.059735 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.059102    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbktj\" (UniqueName: \"kubernetes.io/projected/20db3162-f8b1-44e3-84ad-f149146b0c1a-kube-api-access-bbktj\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.059735 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.059117    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25a060f3-9db2-4020-bacf-6b04c0941a3a-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.059735 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.059130    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.059735 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.059144    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20db3162-f8b1-44e3-84ad-f149146b0c1a-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.059735 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.059158    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.059735 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.059172    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.059735 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.059185    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20db3162-f8b1-44e3-84ad-f149146b0c1a-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.059735 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.059405    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-data" pod "25a060f3-9db2-4020-bacf-6b04c0941a3a" (UID: "25a060f3-9db2-4020-bacf-6b04c0941a3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:33:47.059735 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.059589    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-workload-socket" pod "05470e4a-2ba0-4978-bd4f-99040bd6119d" (UID: "05470e4a-2ba0-4978-bd4f-99040bd6119d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:33:47.060741 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.060715    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a060f3-9db2-4020-bacf-6b04c0941a3a-kube-api-access-xhw5p" pod "25a060f3-9db2-4020-bacf-6b04c0941a3a" (UID: "25a060f3-9db2-4020-bacf-6b04c0941a3a"). InnerVolumeSpecName "kube-api-access-xhw5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:33:47.060886 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.060868    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-envoy" pod "25a060f3-9db2-4020-bacf-6b04c0941a3a" (UID: "25a060f3-9db2-4020-bacf-6b04c0941a3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:33:47.060977 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.060880    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-token" pod "25a060f3-9db2-4020-bacf-6b04c0941a3a" (UID: "25a060f3-9db2-4020-bacf-6b04c0941a3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:33:47.060977 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.060916    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-token" pod "05470e4a-2ba0-4978-bd4f-99040bd6119d" (UID: "05470e4a-2ba0-4978-bd4f-99040bd6119d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:33:47.061499 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.061476    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05470e4a-2ba0-4978-bd4f-99040bd6119d-kube-api-access-9c8nr" pod "05470e4a-2ba0-4978-bd4f-99040bd6119d" (UID: "05470e4a-2ba0-4978-bd4f-99040bd6119d"). InnerVolumeSpecName "kube-api-access-9c8nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:33:47.061928 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.061878    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-podinfo" pod "05470e4a-2ba0-4978-bd4f-99040bd6119d" (UID: "05470e4a-2ba0-4978-bd4f-99040bd6119d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:33:47.062187 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.062161    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-envoy" pod "05470e4a-2ba0-4978-bd4f-99040bd6119d" (UID: "05470e4a-2ba0-4978-bd4f-99040bd6119d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:33:47.062515 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.062487    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-podinfo" pod "25a060f3-9db2-4020-bacf-6b04c0941a3a" (UID: "25a060f3-9db2-4020-bacf-6b04c0941a3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:33:47.160864 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.160833    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhw5p\" (UniqueName: \"kubernetes.io/projected/25a060f3-9db2-4020-bacf-6b04c0941a3a-kube-api-access-xhw5p\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.160864 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.160863    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.161022 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.160875    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.161022 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.160884    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.161022 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.160893    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.161022 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.160901    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05470e4a-2ba0-4978-bd4f-99040bd6119d-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.161022 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.160910    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.161022 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.160919    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c8nr\" (UniqueName: \"kubernetes.io/projected/05470e4a-2ba0-4978-bd4f-99040bd6119d-kube-api-access-9c8nr\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.161022 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.160928    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05470e4a-2ba0-4978-bd4f-99040bd6119d-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.161022 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.160937    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25a060f3-9db2-4020-bacf-6b04c0941a3a-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:47.887819 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.883635    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw" event={"ID":"25a060f3-9db2-4020-bacf-6b04c0941a3a","Type":"ContainerDied","Data":"086ec5bdea47fb2cc7b83c7b2ce73383be259b2df3f517f5dd165216c79f44eb"}
Nov 03 02:33:47.887819 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.883734    5005 scope.go:122] "RemoveContainer" containerID="5ee308942fda5de40f54dc3a653781baf06fdb39225258311a9c5c609e66f259"
Nov 03 02:33:47.887819 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.883913    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"
Nov 03 02:33:47.898418 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.898389    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"
Nov 03 02:33:47.898599 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.898393    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6" event={"ID":"05470e4a-2ba0-4978-bd4f-99040bd6119d","Type":"ContainerDied","Data":"5922836c01b8ae51eafc870044c43d5ff249a2a8dc0262a3ba79aca346a7838f"}
Nov 03 02:33:47.898861 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.898836    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"
Nov 03 02:33:47.972392 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:47.972356    5005 scope.go:122] "RemoveContainer" containerID="80db72b5ba940dc209193d3803a9f2ae5bddd69cb83366b41845752a925885c2"
Nov 03 02:33:48.062869 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:48.062836    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"]
Nov 03 02:33:48.108734 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:48.108680    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-trpcw"]
Nov 03 02:33:48.210884 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:48.210801    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"]
Nov 03 02:33:48.215569 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:48.215537    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-dvw8b"]
Nov 03 02:33:48.317565 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:48.317534    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"]
Nov 03 02:33:48.321197 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:48.321167    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f9bff9-openshift-default-799878f9bf-d4pp6"]
Nov 03 02:33:49.111201 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.111165    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05470e4a-2ba0-4978-bd4f-99040bd6119d" path="/var/lib/kubelet/pods/05470e4a-2ba0-4978-bd4f-99040bd6119d/volumes"
Nov 03 02:33:49.111930 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.111909    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20db3162-f8b1-44e3-84ad-f149146b0c1a" path="/var/lib/kubelet/pods/20db3162-f8b1-44e3-84ad-f149146b0c1a/volumes"
Nov 03 02:33:49.112597 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.112577    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25a060f3-9db2-4020-bacf-6b04c0941a3a" path="/var/lib/kubelet/pods/25a060f3-9db2-4020-bacf-6b04c0941a3a/volumes"
Nov 03 02:33:49.509423 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.509395    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"]
Nov 03 02:33:49.509967 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.509951    5005 cpu_manager.go:414] "containerMap: removing container" podUID="20db3162-f8b1-44e3-84ad-f149146b0c1a" containerName="istio-proxy"
Nov 03 02:33:49.510021 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.509968    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20db3162-f8b1-44e3-84ad-f149146b0c1a" containerName="istio-proxy"
Nov 03 02:33:49.510021 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.509983    5005 cpu_manager.go:414] "containerMap: removing container" podUID="25a060f3-9db2-4020-bacf-6b04c0941a3a" containerName="istio-proxy"
Nov 03 02:33:49.510021 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.509989    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25a060f3-9db2-4020-bacf-6b04c0941a3a" containerName="istio-proxy"
Nov 03 02:33:49.510021 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.510019    5005 cpu_manager.go:414] "containerMap: removing container" podUID="05470e4a-2ba0-4978-bd4f-99040bd6119d" containerName="istio-proxy"
Nov 03 02:33:49.510147 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.510025    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05470e4a-2ba0-4978-bd4f-99040bd6119d" containerName="istio-proxy"
Nov 03 02:33:49.510147 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.510037    5005 cpu_manager.go:414] "containerMap: removing container" podUID="ae2fd3df-1c03-4b71-8b58-0b7532fbd668" containerName="maas-api"
Nov 03 02:33:49.510147 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.510042    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2fd3df-1c03-4b71-8b58-0b7532fbd668" containerName="maas-api"
Nov 03 02:33:49.510147 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.510118    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="20db3162-f8b1-44e3-84ad-f149146b0c1a" containerName="istio-proxy"
Nov 03 02:33:49.510147 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.510129    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="25a060f3-9db2-4020-bacf-6b04c0941a3a" containerName="istio-proxy"
Nov 03 02:33:49.510147 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.510142    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="05470e4a-2ba0-4978-bd4f-99040bd6119d" containerName="istio-proxy"
Nov 03 02:33:49.510327 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.510149    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2fd3df-1c03-4b71-8b58-0b7532fbd668" containerName="maas-api"
Nov 03 02:33:49.515205 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.515187    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"
Nov 03 02:33:49.589813 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.589779    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c21f738-1458-4b56-880f-e4ff7fc8123a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv\" (UID: \"1c21f738-1458-4b56-880f-e4ff7fc8123a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"
Nov 03 02:33:49.589963 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.589826    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c21f738-1458-4b56-880f-e4ff7fc8123a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv\" (UID: \"1c21f738-1458-4b56-880f-e4ff7fc8123a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"
Nov 03 02:33:49.589963 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.589920    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxpgb\" (UniqueName: \"kubernetes.io/projected/1c21f738-1458-4b56-880f-e4ff7fc8123a-kube-api-access-cxpgb\") pod \"maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv\" (UID: \"1c21f738-1458-4b56-880f-e4ff7fc8123a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"
Nov 03 02:33:49.590066 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.590004    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c21f738-1458-4b56-880f-e4ff7fc8123a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv\" (UID: \"1c21f738-1458-4b56-880f-e4ff7fc8123a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"
Nov 03 02:33:49.613550 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.613520    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-3a4683-serving-cert\""
Nov 03 02:33:49.616810 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.616786    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"]
Nov 03 02:33:49.691825 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.691786    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c21f738-1458-4b56-880f-e4ff7fc8123a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv\" (UID: \"1c21f738-1458-4b56-880f-e4ff7fc8123a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"
Nov 03 02:33:49.691825 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.691845    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c21f738-1458-4b56-880f-e4ff7fc8123a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv\" (UID: \"1c21f738-1458-4b56-880f-e4ff7fc8123a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"
Nov 03 02:33:49.692105 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.691894    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxpgb\" (UniqueName: \"kubernetes.io/projected/1c21f738-1458-4b56-880f-e4ff7fc8123a-kube-api-access-cxpgb\") pod \"maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv\" (UID: \"1c21f738-1458-4b56-880f-e4ff7fc8123a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"
Nov 03 02:33:49.692105 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.691921    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c21f738-1458-4b56-880f-e4ff7fc8123a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv\" (UID: \"1c21f738-1458-4b56-880f-e4ff7fc8123a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"
Nov 03 02:33:49.692105 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:33:49.691941    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-serving-cert: secret "maas-api-e2e-shared-a-w1-3a4683-serving-cert" not found
Nov 03 02:33:49.692105 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:33:49.692008    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c21f738-1458-4b56-880f-e4ff7fc8123a-maas-api-tls podName:1c21f738-1458-4b56-880f-e4ff7fc8123a nodeName:}" failed. No retries permitted until 2026-11-03 02:33:50.191990959 +0000 UTC m=+2557.206232303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c21f738-1458-4b56-880f-e4ff7fc8123a-maas-api-tls") pod "maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv" (UID: "1c21f738-1458-4b56-880f-e4ff7fc8123a") : secret "maas-api-e2e-shared-a-w1-3a4683-serving-cert" not found
Nov 03 02:33:49.692660 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.692637    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c21f738-1458-4b56-880f-e4ff7fc8123a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv\" (UID: \"1c21f738-1458-4b56-880f-e4ff7fc8123a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"
Nov 03 02:33:49.694323 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.694305    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c21f738-1458-4b56-880f-e4ff7fc8123a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv\" (UID: \"1c21f738-1458-4b56-880f-e4ff7fc8123a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"
Nov 03 02:33:49.747917 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:49.747882    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxpgb\" (UniqueName: \"kubernetes.io/projected/1c21f738-1458-4b56-880f-e4ff7fc8123a-kube-api-access-cxpgb\") pod \"maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv\" (UID: \"1c21f738-1458-4b56-880f-e4ff7fc8123a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"
Nov 03 02:33:50.205179 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:50.205140    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c21f738-1458-4b56-880f-e4ff7fc8123a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv\" (UID: \"1c21f738-1458-4b56-880f-e4ff7fc8123a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"
Nov 03 02:33:50.207636 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:50.207614    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c21f738-1458-4b56-880f-e4ff7fc8123a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv\" (UID: \"1c21f738-1458-4b56-880f-e4ff7fc8123a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"
Nov 03 02:33:50.429267 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:50.429227    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"
Nov 03 02:33:50.805157 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:50.805115    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"]
Nov 03 02:33:50.811566 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:33:50.811522    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c21f738_1458_4b56_880f_e4ff7fc8123a.slice/crio-b0d58d70c3cb504c02c9f5f2da4e3546cf47aff25575bd9c802429a1b56af265 WatchSource:0}: Error finding container b0d58d70c3cb504c02c9f5f2da4e3546cf47aff25575bd9c802429a1b56af265: Status 404 returned error can't find the container with id b0d58d70c3cb504c02c9f5f2da4e3546cf47aff25575bd9c802429a1b56af265
Nov 03 02:33:50.925457 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:50.925412    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv" event={"ID":"1c21f738-1458-4b56-880f-e4ff7fc8123a","Type":"ContainerStarted","Data":"b0d58d70c3cb504c02c9f5f2da4e3546cf47aff25575bd9c802429a1b56af265"}
Nov 03 02:33:51.934236 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:51.934193    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv" event={"ID":"1c21f738-1458-4b56-880f-e4ff7fc8123a","Type":"ContainerStarted","Data":"59cc5e573ee9c061111eba99a62a1f81e39c273e7c8eb4b5a4632a1909d4cde1"}
Nov 03 02:33:51.934634 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:51.934277    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"
Nov 03 02:33:52.046068 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:52.046019    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv" podStartSLOduration=2.118576326 podStartE2EDuration="3.04600445s" totalImagesPullingTime="927.428124ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:52.044114009 +0000 UTC m=+2559.058355375" watchObservedRunningTime="2026-11-03 02:33:52.04600445 +0000 UTC m=+2559.060245816"
Nov 03 02:33:57.946242 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:33:57.946165    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"
Nov 03 02:34:00.714503 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:34:00.714462    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:34:09.157157 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:09.157119    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"]
Nov 03 02:34:09.161135 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:09.161113    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"
Nov 03 02:34:09.211950 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:09.211914    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-9aebe8-serving-cert\""
Nov 03 02:34:09.261519 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:09.261483    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"]
Nov 03 02:34:09.277005 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:09.276976    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58a3573-fd82-4f92-829b-475591248445-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn\" (UID: \"c58a3573-fd82-4f92-829b-475591248445\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"
Nov 03 02:34:09.277174 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:09.277027    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42pmx\" (UniqueName: \"kubernetes.io/projected/c58a3573-fd82-4f92-829b-475591248445-kube-api-access-42pmx\") pod \"maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn\" (UID: \"c58a3573-fd82-4f92-829b-475591248445\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"
Nov 03 02:34:09.277174 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:09.277056    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58a3573-fd82-4f92-829b-475591248445-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn\" (UID: \"c58a3573-fd82-4f92-829b-475591248445\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"
Nov 03 02:34:09.277268 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:09.277246    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58a3573-fd82-4f92-829b-475591248445-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn\" (UID: \"c58a3573-fd82-4f92-829b-475591248445\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"
Nov 03 02:34:09.378575 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:09.378531    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58a3573-fd82-4f92-829b-475591248445-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn\" (UID: \"c58a3573-fd82-4f92-829b-475591248445\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"
Nov 03 02:34:09.378776 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:09.378631    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58a3573-fd82-4f92-829b-475591248445-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn\" (UID: \"c58a3573-fd82-4f92-829b-475591248445\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"
Nov 03 02:34:09.378776 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:09.378681    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42pmx\" (UniqueName: \"kubernetes.io/projected/c58a3573-fd82-4f92-829b-475591248445-kube-api-access-42pmx\") pod \"maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn\" (UID: \"c58a3573-fd82-4f92-829b-475591248445\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"
Nov 03 02:34:09.378920 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:34:09.378764    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-serving-cert: secret "maas-api-e2e-shared-a-w0-9aebe8-serving-cert" not found
Nov 03 02:34:09.378920 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:34:09.378859    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c58a3573-fd82-4f92-829b-475591248445-maas-api-tls podName:c58a3573-fd82-4f92-829b-475591248445 nodeName:}" failed. No retries permitted until 2026-11-03 02:34:09.878834076 +0000 UTC m=+2576.893075448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c58a3573-fd82-4f92-829b-475591248445-maas-api-tls") pod "maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn" (UID: "c58a3573-fd82-4f92-829b-475591248445") : secret "maas-api-e2e-shared-a-w0-9aebe8-serving-cert" not found
Nov 03 02:34:09.378920 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:09.378771    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58a3573-fd82-4f92-829b-475591248445-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn\" (UID: \"c58a3573-fd82-4f92-829b-475591248445\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"
Nov 03 02:34:09.379419 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:09.379391    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58a3573-fd82-4f92-829b-475591248445-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn\" (UID: \"c58a3573-fd82-4f92-829b-475591248445\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"
Nov 03 02:34:09.381281 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:09.381262    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58a3573-fd82-4f92-829b-475591248445-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn\" (UID: \"c58a3573-fd82-4f92-829b-475591248445\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"
Nov 03 02:34:09.442809 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:09.442661    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42pmx\" (UniqueName: \"kubernetes.io/projected/c58a3573-fd82-4f92-829b-475591248445-kube-api-access-42pmx\") pod \"maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn\" (UID: \"c58a3573-fd82-4f92-829b-475591248445\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"
Nov 03 02:34:09.895180 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:09.895140    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58a3573-fd82-4f92-829b-475591248445-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn\" (UID: \"c58a3573-fd82-4f92-829b-475591248445\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"
Nov 03 02:34:09.897771 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:09.897738    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58a3573-fd82-4f92-829b-475591248445-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn\" (UID: \"c58a3573-fd82-4f92-829b-475591248445\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"
Nov 03 02:34:10.076092 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:10.076057    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"
Nov 03 02:34:10.453599 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:10.453532    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"]
Nov 03 02:34:10.467640 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:34:10.467609    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58a3573_fd82_4f92_829b_475591248445.slice/crio-45d3f5d723654022a9fc8c7ef7f4b53bf6b0d027c001de5d05edbfefbb0fa19a WatchSource:0}: Error finding container 45d3f5d723654022a9fc8c7ef7f4b53bf6b0d027c001de5d05edbfefbb0fa19a: Status 404 returned error can't find the container with id 45d3f5d723654022a9fc8c7ef7f4b53bf6b0d027c001de5d05edbfefbb0fa19a
Nov 03 02:34:11.062724 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:11.062654    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn" event={"ID":"c58a3573-fd82-4f92-829b-475591248445","Type":"ContainerStarted","Data":"45d3f5d723654022a9fc8c7ef7f4b53bf6b0d027c001de5d05edbfefbb0fa19a"}
Nov 03 02:34:12.071569 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:12.071471    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn" event={"ID":"c58a3573-fd82-4f92-829b-475591248445","Type":"ContainerStarted","Data":"2afa0286e14606cbeb8df6667f5fbd80aad43e933bd37d7cfacf1ff3aac60923"}
Nov 03 02:34:12.071962 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:12.071572    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"
Nov 03 02:34:12.179773 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:12.179714    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn" podStartSLOduration=1.86082331 podStartE2EDuration="3.17968162s" totalImagesPullingTime="1.31885831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:34:12.178494474 +0000 UTC m=+2579.192735840" watchObservedRunningTime="2026-11-03 02:34:12.17968162 +0000 UTC m=+2579.193922988"
Nov 03 02:34:18.082876 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:18.082843    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"
Nov 03 02:34:30.713646 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:34:30.713602    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:34:38.191834 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:38.191799    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"]
Nov 03 02:34:38.197104 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:38.197083    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"
Nov 03 02:34:38.292994 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:38.292949    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-f954db-serving-cert\""
Nov 03 02:34:38.296285 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:38.296256    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"]
Nov 03 02:34:38.306764 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:38.306731    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee116929-8ee5-40c2-9384-4fe02d40c720-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f954db-754986b779-j9k2t\" (UID: \"ee116929-8ee5-40c2-9384-4fe02d40c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"
Nov 03 02:34:38.306949 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:38.306924    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlhjl\" (UniqueName: \"kubernetes.io/projected/ee116929-8ee5-40c2-9384-4fe02d40c720-kube-api-access-nlhjl\") pod \"maas-api-e2e-praxis-a-f954db-754986b779-j9k2t\" (UID: \"ee116929-8ee5-40c2-9384-4fe02d40c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"
Nov 03 02:34:38.307047 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:38.307028    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee116929-8ee5-40c2-9384-4fe02d40c720-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f954db-754986b779-j9k2t\" (UID: \"ee116929-8ee5-40c2-9384-4fe02d40c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"
Nov 03 02:34:38.307142 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:38.307128    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee116929-8ee5-40c2-9384-4fe02d40c720-metrics-certs\") pod \"maas-api-e2e-praxis-a-f954db-754986b779-j9k2t\" (UID: \"ee116929-8ee5-40c2-9384-4fe02d40c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"
Nov 03 02:34:38.407909 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:38.407878    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee116929-8ee5-40c2-9384-4fe02d40c720-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f954db-754986b779-j9k2t\" (UID: \"ee116929-8ee5-40c2-9384-4fe02d40c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"
Nov 03 02:34:38.408170 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:38.407950    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee116929-8ee5-40c2-9384-4fe02d40c720-metrics-certs\") pod \"maas-api-e2e-praxis-a-f954db-754986b779-j9k2t\" (UID: \"ee116929-8ee5-40c2-9384-4fe02d40c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"
Nov 03 02:34:38.408170 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:38.408007    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee116929-8ee5-40c2-9384-4fe02d40c720-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f954db-754986b779-j9k2t\" (UID: \"ee116929-8ee5-40c2-9384-4fe02d40c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"
Nov 03 02:34:38.408170 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:34:38.408072    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-serving-cert: secret "maas-api-e2e-praxis-a-f954db-serving-cert" not found
Nov 03 02:34:38.408170 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:38.408126    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlhjl\" (UniqueName: \"kubernetes.io/projected/ee116929-8ee5-40c2-9384-4fe02d40c720-kube-api-access-nlhjl\") pod \"maas-api-e2e-praxis-a-f954db-754986b779-j9k2t\" (UID: \"ee116929-8ee5-40c2-9384-4fe02d40c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"
Nov 03 02:34:38.408170 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:34:38.408167    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee116929-8ee5-40c2-9384-4fe02d40c720-maas-api-tls podName:ee116929-8ee5-40c2-9384-4fe02d40c720 nodeName:}" failed. No retries permitted until 2026-11-03 02:34:38.908145281 +0000 UTC m=+2605.922386639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee116929-8ee5-40c2-9384-4fe02d40c720-maas-api-tls") pod "maas-api-e2e-praxis-a-f954db-754986b779-j9k2t" (UID: "ee116929-8ee5-40c2-9384-4fe02d40c720") : secret "maas-api-e2e-praxis-a-f954db-serving-cert" not found
Nov 03 02:34:38.408678 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:38.408658    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee116929-8ee5-40c2-9384-4fe02d40c720-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f954db-754986b779-j9k2t\" (UID: \"ee116929-8ee5-40c2-9384-4fe02d40c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"
Nov 03 02:34:38.410809 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:38.410788    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee116929-8ee5-40c2-9384-4fe02d40c720-metrics-certs\") pod \"maas-api-e2e-praxis-a-f954db-754986b779-j9k2t\" (UID: \"ee116929-8ee5-40c2-9384-4fe02d40c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"
Nov 03 02:34:38.464305 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:38.464235    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlhjl\" (UniqueName: \"kubernetes.io/projected/ee116929-8ee5-40c2-9384-4fe02d40c720-kube-api-access-nlhjl\") pod \"maas-api-e2e-praxis-a-f954db-754986b779-j9k2t\" (UID: \"ee116929-8ee5-40c2-9384-4fe02d40c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"
Nov 03 02:34:38.915542 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:38.915509    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee116929-8ee5-40c2-9384-4fe02d40c720-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f954db-754986b779-j9k2t\" (UID: \"ee116929-8ee5-40c2-9384-4fe02d40c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"
Nov 03 02:34:38.918145 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:38.918111    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee116929-8ee5-40c2-9384-4fe02d40c720-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f954db-754986b779-j9k2t\" (UID: \"ee116929-8ee5-40c2-9384-4fe02d40c720\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"
Nov 03 02:34:39.111237 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:39.111205    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"
Nov 03 02:34:39.486087 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:39.486003    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"]
Nov 03 02:34:39.492001 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:34:39.491970    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee116929_8ee5_40c2_9384_4fe02d40c720.slice/crio-c3f31783864fee3442a0cb665f2dcbf3493c1921f2ef4ad05f91ab6e0cb49317 WatchSource:0}: Error finding container c3f31783864fee3442a0cb665f2dcbf3493c1921f2ef4ad05f91ab6e0cb49317: Status 404 returned error can't find the container with id c3f31783864fee3442a0cb665f2dcbf3493c1921f2ef4ad05f91ab6e0cb49317
Nov 03 02:34:40.294073 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.294031    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t" event={"ID":"ee116929-8ee5-40c2-9384-4fe02d40c720","Type":"ContainerStarted","Data":"c3f31783864fee3442a0cb665f2dcbf3493c1921f2ef4ad05f91ab6e0cb49317"}
Nov 03 02:34:40.465477 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.465435    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"]
Nov 03 02:34:40.469558 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.469533    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.533687 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.533654    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.533687 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.533688    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.534185 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.533850    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.534185 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.533963    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.534185 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.534091    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.534185 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.534115    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.534185 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.534155    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5zwr\" (UniqueName: \"kubernetes.io/projected/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-kube-api-access-t5zwr\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.534355 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.534265    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.534355 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.534287    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.608470 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.608395    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-f735f4-openshift-default-dockercfg-4w55v\""
Nov 03 02:34:40.635617 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.635576    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.635617 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.635617    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.635903 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.635653    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5zwr\" (UniqueName: \"kubernetes.io/projected/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-kube-api-access-t5zwr\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.635903 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.635686    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.635903 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.635731    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.635903 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.635804    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.635903 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.635832    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.635903 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.635869    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.636236 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.635947    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.636236 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.636135    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.636349 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.636319    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.636484 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.636455    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.636484 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.636479    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.636681 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.636603    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.638340 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.638313    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.638554 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.638538    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.663375 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.663337    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"]
Nov 03 02:34:40.688943 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.688914    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.689244 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.689220    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5zwr\" (UniqueName: \"kubernetes.io/projected/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-kube-api-access-t5zwr\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:40.782743 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:40.782711    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:41.213466 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:41.213429    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"]
Nov 03 02:34:41.219412 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:34:41.219287    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0019a0f_0716_4bd2_be7b_f96ff62bf7d1.slice/crio-41508d2d1e215bedec11301081066c362885a79511e0811b8c3233f374530afc WatchSource:0}: Error finding container 41508d2d1e215bedec11301081066c362885a79511e0811b8c3233f374530afc: Status 404 returned error can't find the container with id 41508d2d1e215bedec11301081066c362885a79511e0811b8c3233f374530afc
Nov 03 02:34:41.222414 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:41.222372    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:34:41.222551 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:41.222446    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:34:41.222551 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:41.222483    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:34:41.315477 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:41.315430    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68" event={"ID":"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1","Type":"ContainerStarted","Data":"d76e0d9841f3f47eb2179c3f908f649313a9c83dcc43f6ca4ec97a1ef489d257"}
Nov 03 02:34:41.315632 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:41.315535    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68" event={"ID":"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1","Type":"ContainerStarted","Data":"41508d2d1e215bedec11301081066c362885a79511e0811b8c3233f374530afc"}
Nov 03 02:34:41.318358 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:41.318327    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t" event={"ID":"ee116929-8ee5-40c2-9384-4fe02d40c720","Type":"ContainerStarted","Data":"f57416d93fb00f71c58361e82737569c2be035bc8984d6118a9f35b8468f3b1d"}
Nov 03 02:34:41.318508 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:41.318457    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"
Nov 03 02:34:41.457834 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:41.457779    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68" podStartSLOduration=1.457763371 podStartE2EDuration="1.457763371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:34:41.453597014 +0000 UTC m=+2608.467838374" watchObservedRunningTime="2026-11-03 02:34:41.457763371 +0000 UTC m=+2608.472004776"
Nov 03 02:34:41.561075 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:41.561005    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t" podStartSLOduration=2.72516209 podStartE2EDuration="3.560986519s" totalImagesPullingTime="835.824429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:34:41.560149762 +0000 UTC m=+2608.574391129" watchObservedRunningTime="2026-11-03 02:34:41.560986519 +0000 UTC m=+2608.575227882"
Nov 03 02:34:41.783727 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:41.783682    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:42.785318 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:42.785284    5005 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 03 02:34:42.785755 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:42.785339    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68" podUID="f0019a0f-0716-4bd2-be7b-f96ff62bf7d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 03 02:34:43.783279 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:43.783239    5005 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 03 02:34:43.783569 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:43.783291    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68" podUID="f0019a0f-0716-4bd2-be7b-f96ff62bf7d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 03 02:34:44.783473 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:44.783436    5005 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 03 02:34:44.783893 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:44.783498    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68" podUID="f0019a0f-0716-4bd2-be7b-f96ff62bf7d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 03 02:34:45.783412 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:45.783363    5005 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 03 02:34:45.783605 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:45.783429    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68" podUID="f0019a0f-0716-4bd2-be7b-f96ff62bf7d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 03 02:34:46.783288 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:46.783251    5005 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 03 02:34:46.783459 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:46.783305    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68" podUID="f0019a0f-0716-4bd2-be7b-f96ff62bf7d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 03 02:34:47.334439 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:47.334404    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"
Nov 03 02:34:47.814232 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:47.812549    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:47.814232 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:47.814165    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:34:47.814633 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:34:47.814612    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 02:35:00.714318 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:35:00.714277    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:35:10.688093 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.688058    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"]
Nov 03 02:35:10.692339 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.692319    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.760125 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.760086    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.760303 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.760176    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.760357 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.760294    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.760397 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.760388    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.760439 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.760428    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.760549 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.760528    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.760595 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.760584    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aef0dd90-1756-409d-91f1-43d598c9bf19-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.760634 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.760614    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khwvx\" (UniqueName: \"kubernetes.io/projected/aef0dd90-1756-409d-91f1-43d598c9bf19-kube-api-access-khwvx\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.760724 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.760712    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.862613 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.862566    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.862825 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.862680    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.862825 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.862759    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.862825 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.862815    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.862984 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.862877    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.862984 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.862925    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.863097 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.862983    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aef0dd90-1756-409d-91f1-43d598c9bf19-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.863097 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.863014    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khwvx\" (UniqueName: \"kubernetes.io/projected/aef0dd90-1756-409d-91f1-43d598c9bf19-kube-api-access-khwvx\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.863097 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.863073    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.863272 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.863158    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.863599 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.863572    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.863599 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.863593    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.863847 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.863822    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.864039 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.863931    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aef0dd90-1756-409d-91f1-43d598c9bf19-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.865475 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.865451    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.865801 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.865779    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.887884 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.887850    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"]
Nov 03 02:35:10.917793 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.917761    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:10.917969 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:10.917830    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khwvx\" (UniqueName: \"kubernetes.io/projected/aef0dd90-1756-409d-91f1-43d598c9bf19-kube-api-access-khwvx\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:11.007265 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:11.007236    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:11.469333 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:11.469237    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"]
Nov 03 02:35:11.475437 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:35:11.475399    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef0dd90_1756_409d_91f1_43d598c9bf19.slice/crio-db3e0e3f2968949d1a795bb86328f5bd868be2cefc73290928e761fa6525a12e WatchSource:0}: Error finding container db3e0e3f2968949d1a795bb86328f5bd868be2cefc73290928e761fa6525a12e: Status 404 returned error can't find the container with id db3e0e3f2968949d1a795bb86328f5bd868be2cefc73290928e761fa6525a12e
Nov 03 02:35:11.477971 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:11.477935    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:35:11.478059 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:11.478024    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:35:11.478059 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:11.478054    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:35:11.536071 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:11.536046    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n" event={"ID":"aef0dd90-1756-409d-91f1-43d598c9bf19","Type":"ContainerStarted","Data":"db3e0e3f2968949d1a795bb86328f5bd868be2cefc73290928e761fa6525a12e"}
Nov 03 02:35:12.567273 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:12.567208    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n" event={"ID":"aef0dd90-1756-409d-91f1-43d598c9bf19","Type":"ContainerStarted","Data":"4c19e2b60a272f10d86ed505257cde1d0d4345532e15aa97517c0b99bd5a98d6"}
Nov 03 02:35:12.775075 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:12.775021    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n" podStartSLOduration=2.7750073090000003 podStartE2EDuration="2.775007309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:12.727471914 +0000 UTC m=+2639.741713294" watchObservedRunningTime="2026-11-03 02:35:12.775007309 +0000 UTC m=+2639.789248675"
Nov 03 02:35:13.020714 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:13.020664    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:13.022949 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:13.022866    5005 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 03 02:35:13.023121 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:13.022965    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n" podUID="aef0dd90-1756-409d-91f1-43d598c9bf19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 03 02:35:14.012080 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:14.012035    5005 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 03 02:35:14.012721 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:14.012668    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n" podUID="aef0dd90-1756-409d-91f1-43d598c9bf19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 03 02:35:15.015726 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:15.015217    5005 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 03 02:35:15.015726 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:15.015286    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n" podUID="aef0dd90-1756-409d-91f1-43d598c9bf19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 03 02:35:16.009163 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:16.009120    5005 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 03 02:35:16.009458 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:16.009431    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n" podUID="aef0dd90-1756-409d-91f1-43d598c9bf19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 03 02:35:17.014171 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:17.014128    5005 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Nov 03 02:35:17.014882 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:17.014842    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n" podUID="aef0dd90-1756-409d-91f1-43d598c9bf19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Nov 03 02:35:18.100540 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:18.100506    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:18.101027 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:18.100969    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:18.103649 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:18.103624    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:35:21.138581 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.138549    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb"]
Nov 03 02:35:21.145529 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.145507    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb"
Nov 03 02:35:21.191315 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.191286    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb"]
Nov 03 02:35:21.199857 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.199824    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/394a32fc-4122-44a7-aef5-937f6f691da2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb\" (UID: \"394a32fc-4122-44a7-aef5-937f6f691da2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb"
Nov 03 02:35:21.200033 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.199967    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p82v\" (UniqueName: \"kubernetes.io/projected/394a32fc-4122-44a7-aef5-937f6f691da2-kube-api-access-9p82v\") pod \"payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb\" (UID: \"394a32fc-4122-44a7-aef5-937f6f691da2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb"
Nov 03 02:35:21.234426 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.234394    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-ffba43-dockercfg-scfgz\""
Nov 03 02:35:21.235330 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.235301    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-ffba43\""
Nov 03 02:35:21.309714 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.301996    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/394a32fc-4122-44a7-aef5-937f6f691da2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb\" (UID: \"394a32fc-4122-44a7-aef5-937f6f691da2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb"
Nov 03 02:35:21.309714 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.302175    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p82v\" (UniqueName: \"kubernetes.io/projected/394a32fc-4122-44a7-aef5-937f6f691da2-kube-api-access-9p82v\") pod \"payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb\" (UID: \"394a32fc-4122-44a7-aef5-937f6f691da2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb"
Nov 03 02:35:21.309714 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.303522    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/394a32fc-4122-44a7-aef5-937f6f691da2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb\" (UID: \"394a32fc-4122-44a7-aef5-937f6f691da2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb"
Nov 03 02:35:21.337721 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.310184    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896"]
Nov 03 02:35:21.342035 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.342000    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896"
Nov 03 02:35:21.351945 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.351873    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896"]
Nov 03 02:35:21.394455 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.394375    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p82v\" (UniqueName: \"kubernetes.io/projected/394a32fc-4122-44a7-aef5-937f6f691da2-kube-api-access-9p82v\") pod \"payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb\" (UID: \"394a32fc-4122-44a7-aef5-937f6f691da2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb"
Nov 03 02:35:21.419203 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.413461    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f6fafc-574b-47e5-b63a-0eed748772ee-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896\" (UID: \"f8f6fafc-574b-47e5-b63a-0eed748772ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896"
Nov 03 02:35:21.419437 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.419421    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6vcm\" (UniqueName: \"kubernetes.io/projected/f8f6fafc-574b-47e5-b63a-0eed748772ee-kube-api-access-h6vcm\") pod \"payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896\" (UID: \"f8f6fafc-574b-47e5-b63a-0eed748772ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896"
Nov 03 02:35:21.459185 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.459155    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb"
Nov 03 02:35:21.520920 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.520886    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f6fafc-574b-47e5-b63a-0eed748772ee-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896\" (UID: \"f8f6fafc-574b-47e5-b63a-0eed748772ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896"
Nov 03 02:35:21.521032 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.520975    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6vcm\" (UniqueName: \"kubernetes.io/projected/f8f6fafc-574b-47e5-b63a-0eed748772ee-kube-api-access-h6vcm\") pod \"payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896\" (UID: \"f8f6fafc-574b-47e5-b63a-0eed748772ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896"
Nov 03 02:35:21.522437 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.522412    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f6fafc-574b-47e5-b63a-0eed748772ee-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896\" (UID: \"f8f6fafc-574b-47e5-b63a-0eed748772ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896"
Nov 03 02:35:21.574850 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.574813    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6vcm\" (UniqueName: \"kubernetes.io/projected/f8f6fafc-574b-47e5-b63a-0eed748772ee-kube-api-access-h6vcm\") pod \"payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896\" (UID: \"f8f6fafc-574b-47e5-b63a-0eed748772ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896"
Nov 03 02:35:21.722998 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.722916    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896"
Nov 03 02:35:21.834979 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:21.833753    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb"]
Nov 03 02:35:21.842048 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:35:21.842008    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394a32fc_4122_44a7_aef5_937f6f691da2.slice/crio-f7aa343ce5c70e4dfc800eab8461baf5257efcc83c371b3ee010488a0e3da248 WatchSource:0}: Error finding container f7aa343ce5c70e4dfc800eab8461baf5257efcc83c371b3ee010488a0e3da248: Status 404 returned error can't find the container with id f7aa343ce5c70e4dfc800eab8461baf5257efcc83c371b3ee010488a0e3da248
Nov 03 02:35:22.108656 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:22.108624    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896"]
Nov 03 02:35:22.113340 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:35:22.113287    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f6fafc_574b_47e5_b63a_0eed748772ee.slice/crio-fbab3d762c1e07c6151fea3847add5724f327e14fa92a976aba8482691601816 WatchSource:0}: Error finding container fbab3d762c1e07c6151fea3847add5724f327e14fa92a976aba8482691601816: Status 404 returned error can't find the container with id fbab3d762c1e07c6151fea3847add5724f327e14fa92a976aba8482691601816
Nov 03 02:35:22.656365 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:22.656332    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb" event={"ID":"394a32fc-4122-44a7-aef5-937f6f691da2","Type":"ContainerStarted","Data":"5ed6c515a6ad98e1bac8e87462dbdfdb4f2a79ab7a0400a06a58d374973dda66"}
Nov 03 02:35:22.656827 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:22.656391    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb" event={"ID":"394a32fc-4122-44a7-aef5-937f6f691da2","Type":"ContainerStarted","Data":"f7aa343ce5c70e4dfc800eab8461baf5257efcc83c371b3ee010488a0e3da248"}
Nov 03 02:35:22.656827 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:22.656532    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb"
Nov 03 02:35:22.658038 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:22.658014    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896" event={"ID":"f8f6fafc-574b-47e5-b63a-0eed748772ee","Type":"ContainerStarted","Data":"a3b388ee209d27862e8dfd4a407ef3d71a8feefbd07f3702f4e5ecabb4074809"}
Nov 03 02:35:22.658143 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:22.658052    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896" event={"ID":"f8f6fafc-574b-47e5-b63a-0eed748772ee","Type":"ContainerStarted","Data":"fbab3d762c1e07c6151fea3847add5724f327e14fa92a976aba8482691601816"}
Nov 03 02:35:22.658143 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:22.658072    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896"
Nov 03 02:35:22.769725 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:22.769600    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb" podStartSLOduration=1.7695861910000001 podStartE2EDuration="1.769586191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:22.767728737 +0000 UTC m=+2649.781970106" watchObservedRunningTime="2026-11-03 02:35:22.769586191 +0000 UTC m=+2649.783827556"
Nov 03 02:35:22.875169 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:22.875112    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896" podStartSLOduration=1.875083228 podStartE2EDuration="1.875083228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:22.87389055 +0000 UTC m=+2649.888131917" watchObservedRunningTime="2026-11-03 02:35:22.875083228 +0000 UTC m=+2649.889324635"
Nov 03 02:35:25.804683 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:25.804647    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"]
Nov 03 02:35:25.809589 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:25.809563    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:25.898538 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:25.898497    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:25.898964 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:25.898934    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:25.899140 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:25.899121    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:25.899281 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:25.899266    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:25.899420 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:25.899405    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:25.899578 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:25.899560    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:25.899747 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:25.899729    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6gmd\" (UniqueName: \"kubernetes.io/projected/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-kube-api-access-l6gmd\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:25.899954 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:25.899935    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:25.900228 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:25.900211    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:25.950715 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:25.945978    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"]
Nov 03 02:35:26.001287 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.001254    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:26.001473 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.001321    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:26.001473 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.001345    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:26.001473 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.001373    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:26.001473 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.001413    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:26.001473 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.001439    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:26.001473 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.001464    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6gmd\" (UniqueName: \"kubernetes.io/projected/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-kube-api-access-l6gmd\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:26.001882 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.001492    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:26.001882 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.001578    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:26.003184 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.003097    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-workload-certs\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:26.003184 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.003128    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-workload-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:26.003361 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.003196    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-data\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:26.003427 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.003361    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-credential-socket\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:26.003582 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.003558    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istiod-ca-cert\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:26.004971 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.004947    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-envoy\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:26.005742 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.005685    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-podinfo\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:26.055734 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.055649    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-token\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:26.056738 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.056717    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6gmd\" (UniqueName: \"kubernetes.io/projected/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-kube-api-access-l6gmd\") pod \"e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") " pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:26.154042 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.153451    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:26.695099 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.695020    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"]
Nov 03 02:35:26.741382 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:35:26.741347    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe7c918_b791_4dd9_bb28_85c79fc9b8c7.slice/crio-5770683220d96351cba53d11abf20d60768f6361340ed7e6dc8ec70d6692e362 WatchSource:0}: Error finding container 5770683220d96351cba53d11abf20d60768f6361340ed7e6dc8ec70d6692e362: Status 404 returned error can't find the container with id 5770683220d96351cba53d11abf20d60768f6361340ed7e6dc8ec70d6692e362
Nov 03 02:35:26.744065 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.744025    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:35:26.744176 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.744100    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:35:26.744176 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:26.744129    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:35:27.692465 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:27.692426    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk" event={"ID":"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7","Type":"ContainerStarted","Data":"2654d77c32f27aea72020987d22b94c234513157f5672e20718d4c847b09ef9d"}
Nov 03 02:35:27.692901 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:27.692505    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk" event={"ID":"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7","Type":"ContainerStarted","Data":"5770683220d96351cba53d11abf20d60768f6361340ed7e6dc8ec70d6692e362"}
Nov 03 02:35:27.850677 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:27.850617    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk" podStartSLOduration=2.850601818 podStartE2EDuration="2.850601818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:27.849373583 +0000 UTC m=+2654.863614964" watchObservedRunningTime="2026-11-03 02:35:27.850601818 +0000 UTC m=+2654.864843183"
Nov 03 02:35:28.166005 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:28.165973    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:28.167513 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:28.167480    5005 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 03 02:35:28.167643 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:28.167528    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk" podUID="1fe7c918-b791-4dd9-bb28-85c79fc9b8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 03 02:35:29.166667 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:29.166632    5005 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 03 02:35:29.167126 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:29.166689    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk" podUID="1fe7c918-b791-4dd9-bb28-85c79fc9b8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 03 02:35:30.166467 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:30.166430    5005 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 03 02:35:30.166632 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:30.166483    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk" podUID="1fe7c918-b791-4dd9-bb28-85c79fc9b8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 03 02:35:30.737252 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:35:30.737216    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:35:31.166642 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:31.166608    5005 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 03 02:35:31.166855 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:31.166662    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk" podUID="1fe7c918-b791-4dd9-bb28-85c79fc9b8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 03 02:35:32.166368 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:32.166333    5005 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 03 02:35:32.166765 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:32.166386    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk" podUID="1fe7c918-b791-4dd9-bb28-85c79fc9b8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 03 02:35:33.170272 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:33.170244    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:33.170687 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:33.170675    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:33.171342 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:33.171321    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:35:33.667543 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:33.667515    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb"
Nov 03 02:35:33.667747 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:33.667582    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896"
Nov 03 02:35:40.750135 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:40.750097    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"]
Nov 03 02:35:40.754498 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:40.754480    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"
Nov 03 02:35:40.804764 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:40.804728    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-7273af-serving-cert\""
Nov 03 02:35:40.809511 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:40.809485    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"]
Nov 03 02:35:40.886286 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:40.886251    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eded8d0f-e8e1-4410-baa9-e2982338f3e3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw\" (UID: \"eded8d0f-e8e1-4410-baa9-e2982338f3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"
Nov 03 02:35:40.886454 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:40.886317    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eded8d0f-e8e1-4410-baa9-e2982338f3e3-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw\" (UID: \"eded8d0f-e8e1-4410-baa9-e2982338f3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"
Nov 03 02:35:40.886454 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:40.886426    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eded8d0f-e8e1-4410-baa9-e2982338f3e3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw\" (UID: \"eded8d0f-e8e1-4410-baa9-e2982338f3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"
Nov 03 02:35:40.886557 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:40.886513    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbm6v\" (UniqueName: \"kubernetes.io/projected/eded8d0f-e8e1-4410-baa9-e2982338f3e3-kube-api-access-sbm6v\") pod \"maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw\" (UID: \"eded8d0f-e8e1-4410-baa9-e2982338f3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"
Nov 03 02:35:40.988230 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:40.988189    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eded8d0f-e8e1-4410-baa9-e2982338f3e3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw\" (UID: \"eded8d0f-e8e1-4410-baa9-e2982338f3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"
Nov 03 02:35:40.988414 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:40.988282    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbm6v\" (UniqueName: \"kubernetes.io/projected/eded8d0f-e8e1-4410-baa9-e2982338f3e3-kube-api-access-sbm6v\") pod \"maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw\" (UID: \"eded8d0f-e8e1-4410-baa9-e2982338f3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"
Nov 03 02:35:40.988498 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:40.988478    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eded8d0f-e8e1-4410-baa9-e2982338f3e3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw\" (UID: \"eded8d0f-e8e1-4410-baa9-e2982338f3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"
Nov 03 02:35:40.988561 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:40.988522    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eded8d0f-e8e1-4410-baa9-e2982338f3e3-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw\" (UID: \"eded8d0f-e8e1-4410-baa9-e2982338f3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"
Nov 03 02:35:40.988662 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:35:40.988638    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-serving-cert: secret "maas-api-e2e-shared-b-w3-7273af-serving-cert" not found
Nov 03 02:35:40.988790 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:35:40.988776    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eded8d0f-e8e1-4410-baa9-e2982338f3e3-maas-api-tls podName:eded8d0f-e8e1-4410-baa9-e2982338f3e3 nodeName:}" failed. No retries permitted until 2026-11-03 02:35:41.48875692 +0000 UTC m=+2668.502998276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eded8d0f-e8e1-4410-baa9-e2982338f3e3-maas-api-tls") pod "maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw" (UID: "eded8d0f-e8e1-4410-baa9-e2982338f3e3") : secret "maas-api-e2e-shared-b-w3-7273af-serving-cert" not found
Nov 03 02:35:40.988963 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:40.988943    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eded8d0f-e8e1-4410-baa9-e2982338f3e3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw\" (UID: \"eded8d0f-e8e1-4410-baa9-e2982338f3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"
Nov 03 02:35:40.991161 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:40.991140    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eded8d0f-e8e1-4410-baa9-e2982338f3e3-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw\" (UID: \"eded8d0f-e8e1-4410-baa9-e2982338f3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"
Nov 03 02:35:41.042021 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:41.041989    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbm6v\" (UniqueName: \"kubernetes.io/projected/eded8d0f-e8e1-4410-baa9-e2982338f3e3-kube-api-access-sbm6v\") pod \"maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw\" (UID: \"eded8d0f-e8e1-4410-baa9-e2982338f3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"
Nov 03 02:35:41.502879 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:41.502852    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eded8d0f-e8e1-4410-baa9-e2982338f3e3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw\" (UID: \"eded8d0f-e8e1-4410-baa9-e2982338f3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"
Nov 03 02:35:41.505264 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:41.505240    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eded8d0f-e8e1-4410-baa9-e2982338f3e3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw\" (UID: \"eded8d0f-e8e1-4410-baa9-e2982338f3e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"
Nov 03 02:35:41.668312 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:41.668267    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"
Nov 03 02:35:42.041117 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:42.041076    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"]
Nov 03 02:35:42.047176 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:35:42.047144    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeded8d0f_e8e1_4410_baa9_e2982338f3e3.slice/crio-9abaebbc8c708a7e4d31fa831411c520447d72b8f08a69746b3d0aef3f1588cc WatchSource:0}: Error finding container 9abaebbc8c708a7e4d31fa831411c520447d72b8f08a69746b3d0aef3f1588cc: Status 404 returned error can't find the container with id 9abaebbc8c708a7e4d31fa831411c520447d72b8f08a69746b3d0aef3f1588cc
Nov 03 02:35:42.048634 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:42.048614    5005 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 02:35:42.820354 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:42.820320    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw" event={"ID":"eded8d0f-e8e1-4410-baa9-e2982338f3e3","Type":"ContainerStarted","Data":"9abaebbc8c708a7e4d31fa831411c520447d72b8f08a69746b3d0aef3f1588cc"}
Nov 03 02:35:43.829939 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:43.829902    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw" event={"ID":"eded8d0f-e8e1-4410-baa9-e2982338f3e3","Type":"ContainerStarted","Data":"846eea1f15d4b0fb2c1dfffac87d0646576570dbfe329d5c36642f8de3893189"}
Nov 03 02:35:43.830388 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:43.829969    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"
Nov 03 02:35:43.947719 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:43.945602    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw" podStartSLOduration=2.898850449 podStartE2EDuration="3.94558142s" totalImagesPullingTime="1.046730971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:43.938909603 +0000 UTC m=+2670.953150970" watchObservedRunningTime="2026-11-03 02:35:43.94558142 +0000 UTC m=+2670.959822786"
Nov 03 02:35:49.841674 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:35:49.841639    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"
Nov 03 02:36:00.714930 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:36:00.714883    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:36:01.273774 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:01.273742    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"]
Nov 03 02:36:01.278101 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:01.278079    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"
Nov 03 02:36:01.307109 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:01.307081    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkgls\" (UniqueName: \"kubernetes.io/projected/a8b19c9c-11e2-4427-8229-86457df5ebfe-kube-api-access-vkgls\") pod \"maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x\" (UID: \"a8b19c9c-11e2-4427-8229-86457df5ebfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"
Nov 03 02:36:01.307239 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:01.307141    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8b19c9c-11e2-4427-8229-86457df5ebfe-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x\" (UID: \"a8b19c9c-11e2-4427-8229-86457df5ebfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"
Nov 03 02:36:01.307283 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:01.307238    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b19c9c-11e2-4427-8229-86457df5ebfe-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x\" (UID: \"a8b19c9c-11e2-4427-8229-86457df5ebfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"
Nov 03 02:36:01.307369 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:01.307355    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b19c9c-11e2-4427-8229-86457df5ebfe-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x\" (UID: \"a8b19c9c-11e2-4427-8229-86457df5ebfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"
Nov 03 02:36:01.373560 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:01.373527    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-78b5c5-serving-cert\""
Nov 03 02:36:01.377523 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:01.377497    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"]
Nov 03 02:36:01.409101 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:01.409070    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b19c9c-11e2-4427-8229-86457df5ebfe-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x\" (UID: \"a8b19c9c-11e2-4427-8229-86457df5ebfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"
Nov 03 02:36:01.409274 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:01.409242    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkgls\" (UniqueName: \"kubernetes.io/projected/a8b19c9c-11e2-4427-8229-86457df5ebfe-kube-api-access-vkgls\") pod \"maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x\" (UID: \"a8b19c9c-11e2-4427-8229-86457df5ebfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"
Nov 03 02:36:01.409343 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:01.409301    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8b19c9c-11e2-4427-8229-86457df5ebfe-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x\" (UID: \"a8b19c9c-11e2-4427-8229-86457df5ebfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"
Nov 03 02:36:01.409401 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:01.409356    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b19c9c-11e2-4427-8229-86457df5ebfe-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x\" (UID: \"a8b19c9c-11e2-4427-8229-86457df5ebfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"
Nov 03 02:36:01.409979 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:36:01.409848    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-serving-cert: secret "maas-api-e2e-shared-b-w1-78b5c5-serving-cert" not found
Nov 03 02:36:01.409979 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:36:01.409931    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8b19c9c-11e2-4427-8229-86457df5ebfe-maas-api-tls podName:a8b19c9c-11e2-4427-8229-86457df5ebfe nodeName:}" failed. No retries permitted until 2026-11-03 02:36:01.909910559 +0000 UTC m=+2688.924151911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a8b19c9c-11e2-4427-8229-86457df5ebfe-maas-api-tls") pod "maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x" (UID: "a8b19c9c-11e2-4427-8229-86457df5ebfe") : secret "maas-api-e2e-shared-b-w1-78b5c5-serving-cert" not found
Nov 03 02:36:01.410221 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:01.410200    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b19c9c-11e2-4427-8229-86457df5ebfe-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x\" (UID: \"a8b19c9c-11e2-4427-8229-86457df5ebfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"
Nov 03 02:36:01.412303 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:01.412276    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b19c9c-11e2-4427-8229-86457df5ebfe-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x\" (UID: \"a8b19c9c-11e2-4427-8229-86457df5ebfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"
Nov 03 02:36:01.463279 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:01.463250    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkgls\" (UniqueName: \"kubernetes.io/projected/a8b19c9c-11e2-4427-8229-86457df5ebfe-kube-api-access-vkgls\") pod \"maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x\" (UID: \"a8b19c9c-11e2-4427-8229-86457df5ebfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"
Nov 03 02:36:01.919591 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:01.919556    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8b19c9c-11e2-4427-8229-86457df5ebfe-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x\" (UID: \"a8b19c9c-11e2-4427-8229-86457df5ebfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"
Nov 03 02:36:01.922061 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:01.922033    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8b19c9c-11e2-4427-8229-86457df5ebfe-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x\" (UID: \"a8b19c9c-11e2-4427-8229-86457df5ebfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"
Nov 03 02:36:02.192363 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:02.192275    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"
Nov 03 02:36:02.566060 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:02.566029    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"]
Nov 03 02:36:02.571126 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:36:02.571090    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b19c9c_11e2_4427_8229_86457df5ebfe.slice/crio-1ffbe088e49c17b7952539c9e1b4e4afd61d9ff20a9d97ec4fdbb89b4eedb01e WatchSource:0}: Error finding container 1ffbe088e49c17b7952539c9e1b4e4afd61d9ff20a9d97ec4fdbb89b4eedb01e: Status 404 returned error can't find the container with id 1ffbe088e49c17b7952539c9e1b4e4afd61d9ff20a9d97ec4fdbb89b4eedb01e
Nov 03 02:36:03.110315 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:03.110281    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x" event={"ID":"a8b19c9c-11e2-4427-8229-86457df5ebfe","Type":"ContainerStarted","Data":"1ffbe088e49c17b7952539c9e1b4e4afd61d9ff20a9d97ec4fdbb89b4eedb01e"}
Nov 03 02:36:04.113552 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:04.113514    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x" event={"ID":"a8b19c9c-11e2-4427-8229-86457df5ebfe","Type":"ContainerStarted","Data":"6b040fae38d603bcbb5af0a634459b8d45f0e55e7988acf55c3ae3ccb828250b"}
Nov 03 02:36:04.114040 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:04.113662    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"
Nov 03 02:36:04.222042 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:04.221978    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x" podStartSLOduration=2.114600695 podStartE2EDuration="3.221959006s" totalImagesPullingTime="1.107358311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:36:04.220919438 +0000 UTC m=+2691.235160805" watchObservedRunningTime="2026-11-03 02:36:04.221959006 +0000 UTC m=+2691.236200387"
Nov 03 02:36:10.126015 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:10.125985    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"
Nov 03 02:36:13.490252 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:13.490221    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:36:13.502468 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:13.502433    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:36:20.353835 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:20.353802    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"]
Nov 03 02:36:20.359804 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:20.359777    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"
Nov 03 02:36:20.389339 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:20.389311    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/008a4c72-1f2f-4fed-ae45-0ab403315d94-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg\" (UID: \"008a4c72-1f2f-4fed-ae45-0ab403315d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"
Nov 03 02:36:20.389489 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:20.389467    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qslrh\" (UniqueName: \"kubernetes.io/projected/008a4c72-1f2f-4fed-ae45-0ab403315d94-kube-api-access-qslrh\") pod \"maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg\" (UID: \"008a4c72-1f2f-4fed-ae45-0ab403315d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"
Nov 03 02:36:20.389560 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:20.389538    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/008a4c72-1f2f-4fed-ae45-0ab403315d94-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg\" (UID: \"008a4c72-1f2f-4fed-ae45-0ab403315d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"
Nov 03 02:36:20.389722 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:20.389688    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/008a4c72-1f2f-4fed-ae45-0ab403315d94-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg\" (UID: \"008a4c72-1f2f-4fed-ae45-0ab403315d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"
Nov 03 02:36:20.411216 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:20.411188    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"]
Nov 03 02:36:20.452893 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:20.452865    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-ffba43-serving-cert\""
Nov 03 02:36:20.491359 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:20.491325    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/008a4c72-1f2f-4fed-ae45-0ab403315d94-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg\" (UID: \"008a4c72-1f2f-4fed-ae45-0ab403315d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"
Nov 03 02:36:20.491532 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:20.491477    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/008a4c72-1f2f-4fed-ae45-0ab403315d94-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg\" (UID: \"008a4c72-1f2f-4fed-ae45-0ab403315d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"
Nov 03 02:36:20.491608 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:20.491532    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/008a4c72-1f2f-4fed-ae45-0ab403315d94-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg\" (UID: \"008a4c72-1f2f-4fed-ae45-0ab403315d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"
Nov 03 02:36:20.491671 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:20.491647    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qslrh\" (UniqueName: \"kubernetes.io/projected/008a4c72-1f2f-4fed-ae45-0ab403315d94-kube-api-access-qslrh\") pod \"maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg\" (UID: \"008a4c72-1f2f-4fed-ae45-0ab403315d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"
Nov 03 02:36:20.491767 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:36:20.491745    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-serving-cert: secret "maas-api-e2e-shared-b-w0-ffba43-serving-cert" not found
Nov 03 02:36:20.491834 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:36:20.491825    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/008a4c72-1f2f-4fed-ae45-0ab403315d94-maas-api-tls podName:008a4c72-1f2f-4fed-ae45-0ab403315d94 nodeName:}" failed. No retries permitted until 2026-11-03 02:36:20.991803728 +0000 UTC m=+2708.006045074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/008a4c72-1f2f-4fed-ae45-0ab403315d94-maas-api-tls") pod "maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg" (UID: "008a4c72-1f2f-4fed-ae45-0ab403315d94") : secret "maas-api-e2e-shared-b-w0-ffba43-serving-cert" not found
Nov 03 02:36:20.493738 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:20.492173    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/008a4c72-1f2f-4fed-ae45-0ab403315d94-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg\" (UID: \"008a4c72-1f2f-4fed-ae45-0ab403315d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"
Nov 03 02:36:20.504178 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:20.504152    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/008a4c72-1f2f-4fed-ae45-0ab403315d94-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg\" (UID: \"008a4c72-1f2f-4fed-ae45-0ab403315d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"
Nov 03 02:36:20.548402 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:20.548365    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qslrh\" (UniqueName: \"kubernetes.io/projected/008a4c72-1f2f-4fed-ae45-0ab403315d94-kube-api-access-qslrh\") pod \"maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg\" (UID: \"008a4c72-1f2f-4fed-ae45-0ab403315d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"
Nov 03 02:36:21.000592 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:21.000541    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/008a4c72-1f2f-4fed-ae45-0ab403315d94-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg\" (UID: \"008a4c72-1f2f-4fed-ae45-0ab403315d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"
Nov 03 02:36:21.003126 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:21.003099    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/008a4c72-1f2f-4fed-ae45-0ab403315d94-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg\" (UID: \"008a4c72-1f2f-4fed-ae45-0ab403315d94\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"
Nov 03 02:36:21.274361 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:21.274334    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"
Nov 03 02:36:21.655350 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:21.655260    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"]
Nov 03 02:36:21.660421 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:36:21.660392    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008a4c72_1f2f_4fed_ae45_0ab403315d94.slice/crio-1c918cbea6dc6f81d576f8fe2c355ebe898a57e8385f295b114061e42aaa4fcd WatchSource:0}: Error finding container 1c918cbea6dc6f81d576f8fe2c355ebe898a57e8385f295b114061e42aaa4fcd: Status 404 returned error can't find the container with id 1c918cbea6dc6f81d576f8fe2c355ebe898a57e8385f295b114061e42aaa4fcd
Nov 03 02:36:22.238101 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:22.238065    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg" event={"ID":"008a4c72-1f2f-4fed-ae45-0ab403315d94","Type":"ContainerStarted","Data":"1c918cbea6dc6f81d576f8fe2c355ebe898a57e8385f295b114061e42aaa4fcd"}
Nov 03 02:36:23.246888 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:23.246852    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg" event={"ID":"008a4c72-1f2f-4fed-ae45-0ab403315d94","Type":"ContainerStarted","Data":"eb1e468ebf8d2bb7871c23c53759eac27e5564d841be196150b72a93458faa58"}
Nov 03 02:36:23.247355 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:23.246945    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"
Nov 03 02:36:23.382848 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:23.382777    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg" podStartSLOduration=2.536419154 podStartE2EDuration="3.382675873s" totalImagesPullingTime="846.256719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:36:23.358096026 +0000 UTC m=+2710.372337394" watchObservedRunningTime="2026-11-03 02:36:23.382675873 +0000 UTC m=+2710.396917239"
Nov 03 02:36:29.260671 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:29.260640    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"
Nov 03 02:36:29.936900 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:29.936864    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"]
Nov 03 02:36:29.941365 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:29.941345    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"
Nov 03 02:36:29.991580 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:29.991543    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-957b33-serving-cert\""
Nov 03 02:36:29.996445 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:29.996420    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"]
Nov 03 02:36:30.019303 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:30.019268    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5618cf05-a244-4531-9eca-ee70c484da3e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9\" (UID: \"5618cf05-a244-4531-9eca-ee70c484da3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"
Nov 03 02:36:30.019796 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:30.019387    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5618cf05-a244-4531-9eca-ee70c484da3e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9\" (UID: \"5618cf05-a244-4531-9eca-ee70c484da3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"
Nov 03 02:36:30.019796 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:30.019459    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lws4p\" (UniqueName: \"kubernetes.io/projected/5618cf05-a244-4531-9eca-ee70c484da3e-kube-api-access-lws4p\") pod \"maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9\" (UID: \"5618cf05-a244-4531-9eca-ee70c484da3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"
Nov 03 02:36:30.019796 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:30.019518    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5618cf05-a244-4531-9eca-ee70c484da3e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9\" (UID: \"5618cf05-a244-4531-9eca-ee70c484da3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"
Nov 03 02:36:30.121215 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:30.121180    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5618cf05-a244-4531-9eca-ee70c484da3e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9\" (UID: \"5618cf05-a244-4531-9eca-ee70c484da3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"
Nov 03 02:36:30.121422 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:30.121297    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5618cf05-a244-4531-9eca-ee70c484da3e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9\" (UID: \"5618cf05-a244-4531-9eca-ee70c484da3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"
Nov 03 02:36:30.121422 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:30.121373    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lws4p\" (UniqueName: \"kubernetes.io/projected/5618cf05-a244-4531-9eca-ee70c484da3e-kube-api-access-lws4p\") pod \"maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9\" (UID: \"5618cf05-a244-4531-9eca-ee70c484da3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"
Nov 03 02:36:30.121595 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:30.121431    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5618cf05-a244-4531-9eca-ee70c484da3e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9\" (UID: \"5618cf05-a244-4531-9eca-ee70c484da3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"
Nov 03 02:36:30.121595 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:36:30.121554    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-serving-cert: secret "maas-api-e2e-shared-b-w2-957b33-serving-cert" not found
Nov 03 02:36:30.121762 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:36:30.121638    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5618cf05-a244-4531-9eca-ee70c484da3e-maas-api-tls podName:5618cf05-a244-4531-9eca-ee70c484da3e nodeName:}" failed. No retries permitted until 2026-11-03 02:36:30.621616094 +0000 UTC m=+2717.635857439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5618cf05-a244-4531-9eca-ee70c484da3e-maas-api-tls") pod "maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9" (UID: "5618cf05-a244-4531-9eca-ee70c484da3e") : secret "maas-api-e2e-shared-b-w2-957b33-serving-cert" not found
Nov 03 02:36:30.122386 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:30.122352    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5618cf05-a244-4531-9eca-ee70c484da3e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9\" (UID: \"5618cf05-a244-4531-9eca-ee70c484da3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"
Nov 03 02:36:30.124544 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:30.124517    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5618cf05-a244-4531-9eca-ee70c484da3e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9\" (UID: \"5618cf05-a244-4531-9eca-ee70c484da3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"
Nov 03 02:36:30.175772 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:30.175743    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lws4p\" (UniqueName: \"kubernetes.io/projected/5618cf05-a244-4531-9eca-ee70c484da3e-kube-api-access-lws4p\") pod \"maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9\" (UID: \"5618cf05-a244-4531-9eca-ee70c484da3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"
Nov 03 02:36:30.630750 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:30.630714    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5618cf05-a244-4531-9eca-ee70c484da3e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9\" (UID: \"5618cf05-a244-4531-9eca-ee70c484da3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"
Nov 03 02:36:30.633103 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:30.633076    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5618cf05-a244-4531-9eca-ee70c484da3e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9\" (UID: \"5618cf05-a244-4531-9eca-ee70c484da3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"
Nov 03 02:36:30.714197 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:36:30.714154    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:36:30.856441 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:30.856408    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"
Nov 03 02:36:31.229198 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:31.229118    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"]
Nov 03 02:36:31.234568 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:36:31.234545    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5618cf05_a244_4531_9eca_ee70c484da3e.slice/crio-a32108c6e9bbc096bfe05c6a09bb50e2650a8df10bbaa8141996f201b7b5072c WatchSource:0}: Error finding container a32108c6e9bbc096bfe05c6a09bb50e2650a8df10bbaa8141996f201b7b5072c: Status 404 returned error can't find the container with id a32108c6e9bbc096bfe05c6a09bb50e2650a8df10bbaa8141996f201b7b5072c
Nov 03 02:36:31.302660 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:31.302626    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9" event={"ID":"5618cf05-a244-4531-9eca-ee70c484da3e","Type":"ContainerStarted","Data":"a32108c6e9bbc096bfe05c6a09bb50e2650a8df10bbaa8141996f201b7b5072c"}
Nov 03 02:36:32.311995 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:32.311951    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9" event={"ID":"5618cf05-a244-4531-9eca-ee70c484da3e","Type":"ContainerStarted","Data":"3d0ccbb607822f8f459c3a6e646c849dcef8feeebec3784c44058f4e6a058137"}
Nov 03 02:36:32.312470 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:32.312023    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"
Nov 03 02:36:32.421400 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:32.421306    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9" podStartSLOduration=2.555348086 podStartE2EDuration="3.42129097s" totalImagesPullingTime="865.942884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:36:32.418422417 +0000 UTC m=+2719.432663784" watchObservedRunningTime="2026-11-03 02:36:32.42129097 +0000 UTC m=+2719.435532336"
Nov 03 02:36:38.326399 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:36:38.326368    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"
Nov 03 02:37:00.714223 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:37:00.714138    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:37:03.646220 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:03.646186    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"]
Nov 03 02:37:03.652097 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:03.652068    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"
Nov 03 02:37:03.706047 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:03.706016    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"]
Nov 03 02:37:03.747108 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:03.747076    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-4616ec-serving-cert\""
Nov 03 02:37:03.816684 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:03.816654    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ql4l\" (UniqueName: \"kubernetes.io/projected/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-kube-api-access-6ql4l\") pod \"maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th\" (UID: \"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"
Nov 03 02:37:03.816863 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:03.816706    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th\" (UID: \"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"
Nov 03 02:37:03.816962 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:03.816934    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th\" (UID: \"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"
Nov 03 02:37:03.817068 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:03.817052    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-metrics-certs\") pod \"maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th\" (UID: \"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"
Nov 03 02:37:03.929022 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:03.928928    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th\" (UID: \"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"
Nov 03 02:37:03.938798 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:03.938758    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-metrics-certs\") pod \"maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th\" (UID: \"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"
Nov 03 02:37:03.939290 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:03.939228    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ql4l\" (UniqueName: \"kubernetes.io/projected/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-kube-api-access-6ql4l\") pod \"maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th\" (UID: \"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"
Nov 03 02:37:03.939484 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:03.939469    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th\" (UID: \"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"
Nov 03 02:37:03.940095 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:37:03.939971    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-serving-cert: secret "maas-api-e2e-praxis-b-4616ec-serving-cert" not found
Nov 03 02:37:03.940313 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:37:03.940302    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-maas-api-tls podName:f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da nodeName:}" failed. No retries permitted until 2026-11-03 02:37:04.440281237 +0000 UTC m=+2751.454522586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-maas-api-tls") pod "maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th" (UID: "f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da") : secret "maas-api-e2e-praxis-b-4616ec-serving-cert" not found
Nov 03 02:37:03.940972 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:03.937837    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th\" (UID: \"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"
Nov 03 02:37:03.971975 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:03.971938    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-metrics-certs\") pod \"maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th\" (UID: \"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"
Nov 03 02:37:03.994844 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:03.994814    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ql4l\" (UniqueName: \"kubernetes.io/projected/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-kube-api-access-6ql4l\") pod \"maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th\" (UID: \"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"
Nov 03 02:37:04.446432 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:04.446395    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th\" (UID: \"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"
Nov 03 02:37:04.448882 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:04.448857    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th\" (UID: \"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"
Nov 03 02:37:04.565936 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:04.565896    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"
Nov 03 02:37:04.944316 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:04.944256    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"]
Nov 03 02:37:04.949953 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:37:04.949891    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73f1f0f_6ffa_4ed7_b8cb_b578f56f58da.slice/crio-887c792ba801e69fcfd5f2ee78216587ae91f5ce3e8b1885bac620b5ae7221e8 WatchSource:0}: Error finding container 887c792ba801e69fcfd5f2ee78216587ae91f5ce3e8b1885bac620b5ae7221e8: Status 404 returned error can't find the container with id 887c792ba801e69fcfd5f2ee78216587ae91f5ce3e8b1885bac620b5ae7221e8
Nov 03 02:37:05.532411 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:05.532373    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th" event={"ID":"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da","Type":"ContainerStarted","Data":"887c792ba801e69fcfd5f2ee78216587ae91f5ce3e8b1885bac620b5ae7221e8"}
Nov 03 02:37:06.541573 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:06.541529    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th" event={"ID":"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da","Type":"ContainerStarted","Data":"7310319571d8fb5f4167b9b2d22515cac200811c6cf944b7453c197484bcd4d7"}
Nov 03 02:37:06.542122 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:06.541601    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"
Nov 03 02:37:12.554535 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:12.554502    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"
Nov 03 02:37:12.663483 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:37:12.663427    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th" podStartSLOduration=8.671065372 podStartE2EDuration="9.663413832s" totalImagesPullingTime="992.34846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:37:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:37:06.672548345 +0000 UTC m=+2753.686789712" watchObservedRunningTime="2026-11-03 02:37:12.663413832 +0000 UTC m=+2759.677655197"
Nov 03 02:37:30.714544 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:37:30.714498    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:38:00.715820 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:38:00.715779    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:38:13.860429 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:38:13.860392    5005 scope.go:122] "RemoveContainer" containerID="d643465b087ddaf852368fe9d87dda5934e16a8e9b9e8fc333a0daba254e61e4"
Nov 03 02:38:30.715738 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:38:30.715622    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:38:59.445899 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:38:59.445863    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"]
Nov 03 02:38:59.446661 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:38:59.446093    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h" podUID="5b97ff37-eae3-4020-86ed-71477662e225" containerName="istio-proxy" containerID="cri-o://85653861162e40bd77747f682139ef697c7c547a0cc35ae1ac652629c9b08633" gracePeriod=30
Nov 03 02:39:00.715665 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:39:00.715630    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:39:04.718218 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.718149    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:39:04.882085 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.882046    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-credential-socket\") pod \"5b97ff37-eae3-4020-86ed-71477662e225\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") "
Nov 03 02:39:04.882085 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.882101    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-istio-envoy\") pod \"5b97ff37-eae3-4020-86ed-71477662e225\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") "
Nov 03 02:39:04.882349 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.882135    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b97ff37-eae3-4020-86ed-71477662e225-kube-api-access-bqr8q\" (UniqueName: \"kubernetes.io/projected/5b97ff37-eae3-4020-86ed-71477662e225-kube-api-access-bqr8q\") pod \"5b97ff37-eae3-4020-86ed-71477662e225\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") "
Nov 03 02:39:04.882349 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.882306    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-workload-socket\") pod \"5b97ff37-eae3-4020-86ed-71477662e225\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") "
Nov 03 02:39:04.882513 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.882407    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-istio-data\") pod \"5b97ff37-eae3-4020-86ed-71477662e225\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") "
Nov 03 02:39:04.882859 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.882680    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-istio-data" pod "5b97ff37-eae3-4020-86ed-71477662e225" (UID: "5b97ff37-eae3-4020-86ed-71477662e225"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:04.882992 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.882887    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-credential-socket" pod "5b97ff37-eae3-4020-86ed-71477662e225" (UID: "5b97ff37-eae3-4020-86ed-71477662e225"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:04.882992 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.882957    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b97ff37-eae3-4020-86ed-71477662e225-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b97ff37-eae3-4020-86ed-71477662e225-istiod-ca-cert\") pod \"5b97ff37-eae3-4020-86ed-71477662e225\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") "
Nov 03 02:39:04.883117 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.883016    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-workload-certs\") pod \"5b97ff37-eae3-4020-86ed-71477662e225\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") "
Nov 03 02:39:04.883117 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.883071    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b97ff37-eae3-4020-86ed-71477662e225-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b97ff37-eae3-4020-86ed-71477662e225-istio-podinfo\") pod \"5b97ff37-eae3-4020-86ed-71477662e225\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") "
Nov 03 02:39:04.883215 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.883190    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b97ff37-eae3-4020-86ed-71477662e225-istio-token\" (UniqueName: \"kubernetes.io/projected/5b97ff37-eae3-4020-86ed-71477662e225-istio-token\") pod \"5b97ff37-eae3-4020-86ed-71477662e225\" (UID: \"5b97ff37-eae3-4020-86ed-71477662e225\") "
Nov 03 02:39:04.883515 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.883311    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-workload-socket" pod "5b97ff37-eae3-4020-86ed-71477662e225" (UID: "5b97ff37-eae3-4020-86ed-71477662e225"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:04.884383 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.883790    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b97ff37-eae3-4020-86ed-71477662e225-istiod-ca-cert" pod "5b97ff37-eae3-4020-86ed-71477662e225" (UID: "5b97ff37-eae3-4020-86ed-71477662e225"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:39:04.884383 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.884298    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:04.884383 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.884324    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:04.884383 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.884339    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b97ff37-eae3-4020-86ed-71477662e225-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:04.884383 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.884354    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:04.884853 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.884821    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-workload-certs" pod "5b97ff37-eae3-4020-86ed-71477662e225" (UID: "5b97ff37-eae3-4020-86ed-71477662e225"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:04.886092 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.885871    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-istio-envoy" pod "5b97ff37-eae3-4020-86ed-71477662e225" (UID: "5b97ff37-eae3-4020-86ed-71477662e225"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:04.887065 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.887025    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b97ff37-eae3-4020-86ed-71477662e225-istio-podinfo" pod "5b97ff37-eae3-4020-86ed-71477662e225" (UID: "5b97ff37-eae3-4020-86ed-71477662e225"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:39:04.887178 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.887082    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b97ff37-eae3-4020-86ed-71477662e225-kube-api-access-bqr8q" pod "5b97ff37-eae3-4020-86ed-71477662e225" (UID: "5b97ff37-eae3-4020-86ed-71477662e225"). InnerVolumeSpecName "kube-api-access-bqr8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:39:04.888287 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.888265    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b97ff37-eae3-4020-86ed-71477662e225-istio-token" pod "5b97ff37-eae3-4020-86ed-71477662e225" (UID: "5b97ff37-eae3-4020-86ed-71477662e225"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:39:04.985211 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.985132    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b97ff37-eae3-4020-86ed-71477662e225-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:04.985211 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.985163    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:04.985211 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.985173    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqr8q\" (UniqueName: \"kubernetes.io/projected/5b97ff37-eae3-4020-86ed-71477662e225-kube-api-access-bqr8q\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:04.985211 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.985185    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b97ff37-eae3-4020-86ed-71477662e225-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:04.985211 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:04.985193    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b97ff37-eae3-4020-86ed-71477662e225-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:05.389984 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:05.389949    5005 generic.go:356] "Generic (PLEG): container finished" podID="5b97ff37-eae3-4020-86ed-71477662e225" containerID="85653861162e40bd77747f682139ef697c7c547a0cc35ae1ac652629c9b08633" exitCode=0
Nov 03 02:39:05.390168 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:05.390033    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"
Nov 03 02:39:05.390168 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:05.390035    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h" event={"ID":"5b97ff37-eae3-4020-86ed-71477662e225","Type":"ContainerDied","Data":"85653861162e40bd77747f682139ef697c7c547a0cc35ae1ac652629c9b08633"}
Nov 03 02:39:05.390168 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:05.390131    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h" event={"ID":"5b97ff37-eae3-4020-86ed-71477662e225","Type":"ContainerDied","Data":"326e15003073dabdbbe77df276741d99280e35079336707a4789e88e7e0e9cf1"}
Nov 03 02:39:05.390168 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:05.390159    5005 scope.go:122] "RemoveContainer" containerID="85653861162e40bd77747f682139ef697c7c547a0cc35ae1ac652629c9b08633"
Nov 03 02:39:05.400784 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:05.400760    5005 scope.go:122] "RemoveContainer" containerID="85653861162e40bd77747f682139ef697c7c547a0cc35ae1ac652629c9b08633"
Nov 03 02:39:05.401038 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:39:05.401016    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85653861162e40bd77747f682139ef697c7c547a0cc35ae1ac652629c9b08633\": container with ID starting with 85653861162e40bd77747f682139ef697c7c547a0cc35ae1ac652629c9b08633 not found: ID does not exist" containerID="85653861162e40bd77747f682139ef697c7c547a0cc35ae1ac652629c9b08633"
Nov 03 02:39:05.401096 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:05.401048    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85653861162e40bd77747f682139ef697c7c547a0cc35ae1ac652629c9b08633"} err="failed to get container status \"85653861162e40bd77747f682139ef697c7c547a0cc35ae1ac652629c9b08633\": rpc error: code = NotFound desc = could not find container \"85653861162e40bd77747f682139ef697c7c547a0cc35ae1ac652629c9b08633\": container with ID starting with 85653861162e40bd77747f682139ef697c7c547a0cc35ae1ac652629c9b08633 not found: ID does not exist"
Nov 03 02:39:05.545651 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:05.545621    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"]
Nov 03 02:39:05.591711 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:05.591670    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-m894h"]
Nov 03 02:39:07.110926 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:07.110876    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b97ff37-eae3-4020-86ed-71477662e225" path="/var/lib/kubelet/pods/5b97ff37-eae3-4020-86ed-71477662e225/volumes"
Nov 03 02:39:09.172198 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:09.172164    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"]
Nov 03 02:39:09.172709 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:09.172476    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr" podUID="1fc5f95b-3ced-444a-aab9-39c666ac1a29" containerName="istio-proxy" containerID="cri-o://89e3c0b1ac2138e9f48f3aa6e80a49669499420a7cb6592661da37aaf1cdbc1f" gracePeriod=30
Nov 03 02:39:14.451935 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.451874    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:39:14.468919 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.468886    5005 generic.go:356] "Generic (PLEG): container finished" podID="1fc5f95b-3ced-444a-aab9-39c666ac1a29" containerID="89e3c0b1ac2138e9f48f3aa6e80a49669499420a7cb6592661da37aaf1cdbc1f" exitCode=0
Nov 03 02:39:14.469037 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.468932    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"
Nov 03 02:39:14.469037 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.468965    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr" event={"ID":"1fc5f95b-3ced-444a-aab9-39c666ac1a29","Type":"ContainerDied","Data":"89e3c0b1ac2138e9f48f3aa6e80a49669499420a7cb6592661da37aaf1cdbc1f"}
Nov 03 02:39:14.469037 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.469030    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr" event={"ID":"1fc5f95b-3ced-444a-aab9-39c666ac1a29","Type":"ContainerDied","Data":"d77bb10d12f35284cc3dfdcb0cc143c202b888b9e08ff24df4afd49a78de3ebc"}
Nov 03 02:39:14.469152 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.469061    5005 scope.go:122] "RemoveContainer" containerID="89e3c0b1ac2138e9f48f3aa6e80a49669499420a7cb6592661da37aaf1cdbc1f"
Nov 03 02:39:14.479874 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.479855    5005 scope.go:122] "RemoveContainer" containerID="89e3c0b1ac2138e9f48f3aa6e80a49669499420a7cb6592661da37aaf1cdbc1f"
Nov 03 02:39:14.480125 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:39:14.480105    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e3c0b1ac2138e9f48f3aa6e80a49669499420a7cb6592661da37aaf1cdbc1f\": container with ID starting with 89e3c0b1ac2138e9f48f3aa6e80a49669499420a7cb6592661da37aaf1cdbc1f not found: ID does not exist" containerID="89e3c0b1ac2138e9f48f3aa6e80a49669499420a7cb6592661da37aaf1cdbc1f"
Nov 03 02:39:14.480192 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.480139    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e3c0b1ac2138e9f48f3aa6e80a49669499420a7cb6592661da37aaf1cdbc1f"} err="failed to get container status \"89e3c0b1ac2138e9f48f3aa6e80a49669499420a7cb6592661da37aaf1cdbc1f\": rpc error: code = NotFound desc = could not find container \"89e3c0b1ac2138e9f48f3aa6e80a49669499420a7cb6592661da37aaf1cdbc1f\": container with ID starting with 89e3c0b1ac2138e9f48f3aa6e80a49669499420a7cb6592661da37aaf1cdbc1f not found: ID does not exist"
Nov 03 02:39:14.535908 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.535884    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-data\") pod \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") "
Nov 03 02:39:14.536047 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.535953    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-workload-certs\") pod \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") "
Nov 03 02:39:14.536047 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.535976    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istiod-ca-cert\") pod \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") "
Nov 03 02:39:14.536047 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.535994    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-credential-socket\") pod \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") "
Nov 03 02:39:14.536047 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.536013    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-envoy\") pod \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") "
Nov 03 02:39:14.536047 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.536046    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-token\" (UniqueName: \"kubernetes.io/projected/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-token\") pod \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") "
Nov 03 02:39:14.536365 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.536156    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-workload-socket\") pod \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") "
Nov 03 02:39:14.536365 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.536185    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-data" pod "1fc5f95b-3ced-444a-aab9-39c666ac1a29" (UID: "1fc5f95b-3ced-444a-aab9-39c666ac1a29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:14.536365 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.536226    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-podinfo\") pod \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") "
Nov 03 02:39:14.536365 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.536293    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fc5f95b-3ced-444a-aab9-39c666ac1a29-kube-api-access-7xjvs\" (UniqueName: \"kubernetes.io/projected/1fc5f95b-3ced-444a-aab9-39c666ac1a29-kube-api-access-7xjvs\") pod \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\" (UID: \"1fc5f95b-3ced-444a-aab9-39c666ac1a29\") "
Nov 03 02:39:14.536365 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.536298    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-credential-socket" pod "1fc5f95b-3ced-444a-aab9-39c666ac1a29" (UID: "1fc5f95b-3ced-444a-aab9-39c666ac1a29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:14.536365 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.536336    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-workload-certs" pod "1fc5f95b-3ced-444a-aab9-39c666ac1a29" (UID: "1fc5f95b-3ced-444a-aab9-39c666ac1a29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:14.536707 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.536396    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-workload-socket" pod "1fc5f95b-3ced-444a-aab9-39c666ac1a29" (UID: "1fc5f95b-3ced-444a-aab9-39c666ac1a29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:14.536707 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.536412    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istiod-ca-cert" pod "1fc5f95b-3ced-444a-aab9-39c666ac1a29" (UID: "1fc5f95b-3ced-444a-aab9-39c666ac1a29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:39:14.536947 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.536926    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:14.537021 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.536952    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:14.537021 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.536969    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:14.537021 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.536985    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:14.537021 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.536999    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:14.538418 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.538384    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-envoy" pod "1fc5f95b-3ced-444a-aab9-39c666ac1a29" (UID: "1fc5f95b-3ced-444a-aab9-39c666ac1a29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:14.538509 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.538428    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-token" pod "1fc5f95b-3ced-444a-aab9-39c666ac1a29" (UID: "1fc5f95b-3ced-444a-aab9-39c666ac1a29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:39:14.538509 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.538467    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-podinfo" pod "1fc5f95b-3ced-444a-aab9-39c666ac1a29" (UID: "1fc5f95b-3ced-444a-aab9-39c666ac1a29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:39:14.538638 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.538574    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc5f95b-3ced-444a-aab9-39c666ac1a29-kube-api-access-7xjvs" pod "1fc5f95b-3ced-444a-aab9-39c666ac1a29" (UID: "1fc5f95b-3ced-444a-aab9-39c666ac1a29"). InnerVolumeSpecName "kube-api-access-7xjvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:39:14.638274 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.638240    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:14.638274 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.638274    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:14.638461 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.638291    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc5f95b-3ced-444a-aab9-39c666ac1a29-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:14.638461 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.638307    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xjvs\" (UniqueName: \"kubernetes.io/projected/1fc5f95b-3ced-444a-aab9-39c666ac1a29-kube-api-access-7xjvs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:14.885345 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.885297    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"]
Nov 03 02:39:14.888879 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:14.888851    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-tnlbr"]
Nov 03 02:39:15.111453 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:39:15.111376    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fc5f95b-3ced-444a-aab9-39c666ac1a29" path="/var/lib/kubelet/pods/1fc5f95b-3ced-444a-aab9-39c666ac1a29/volumes"
Nov 03 02:39:30.714216 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:39:30.714173    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:40:00.714046 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:40:00.713954    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:40:30.716545 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:40:30.716506    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:41:00.714305 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:41:00.714268    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:41:13.538988 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:13.538957    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:41:13.550042 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:13.550018    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:41:25.114082 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:25.113996    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"]
Nov 03 02:41:25.114558 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:25.114197    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr" podUID="ecdc91da-4a69-4e9a-b46c-12d9cf909aa2" containerName="istio-proxy" containerID="cri-o://517e2513e9d6e615828d0d8e436e38a27780ba50b6c6acb43e349c0857e159eb" gracePeriod=30
Nov 03 02:41:30.146355 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.146323    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 02:41:30.146682 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.146389    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr" podUID="ecdc91da-4a69-4e9a-b46c-12d9cf909aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 02:41:30.382964 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.382892    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:41:30.473190 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.473152    5005 generic.go:356] "Generic (PLEG): container finished" podID="ecdc91da-4a69-4e9a-b46c-12d9cf909aa2" containerID="517e2513e9d6e615828d0d8e436e38a27780ba50b6c6acb43e349c0857e159eb" exitCode=0
Nov 03 02:41:30.473365 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.473204    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"
Nov 03 02:41:30.473365 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.473233    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr" event={"ID":"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2","Type":"ContainerDied","Data":"517e2513e9d6e615828d0d8e436e38a27780ba50b6c6acb43e349c0857e159eb"}
Nov 03 02:41:30.473365 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.473290    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr" event={"ID":"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2","Type":"ContainerDied","Data":"9ae55b5d3e511540cff286d4708e1191389038858bec738ede46440f71ecb0ee"}
Nov 03 02:41:30.473365 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.473315    5005 scope.go:122] "RemoveContainer" containerID="517e2513e9d6e615828d0d8e436e38a27780ba50b6c6acb43e349c0857e159eb"
Nov 03 02:41:30.484960 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.484937    5005 scope.go:122] "RemoveContainer" containerID="517e2513e9d6e615828d0d8e436e38a27780ba50b6c6acb43e349c0857e159eb"
Nov 03 02:41:30.485287 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:41:30.485265    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"517e2513e9d6e615828d0d8e436e38a27780ba50b6c6acb43e349c0857e159eb\": container with ID starting with 517e2513e9d6e615828d0d8e436e38a27780ba50b6c6acb43e349c0857e159eb not found: ID does not exist" containerID="517e2513e9d6e615828d0d8e436e38a27780ba50b6c6acb43e349c0857e159eb"
Nov 03 02:41:30.485359 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.485296    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"517e2513e9d6e615828d0d8e436e38a27780ba50b6c6acb43e349c0857e159eb"} err="failed to get container status \"517e2513e9d6e615828d0d8e436e38a27780ba50b6c6acb43e349c0857e159eb\": rpc error: code = NotFound desc = could not find container \"517e2513e9d6e615828d0d8e436e38a27780ba50b6c6acb43e349c0857e159eb\": container with ID starting with 517e2513e9d6e615828d0d8e436e38a27780ba50b6c6acb43e349c0857e159eb not found: ID does not exist"
Nov 03 02:41:30.521389 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.521352    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-credential-socket\") pod \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") "
Nov 03 02:41:30.521590 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.521457    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-workload-certs\") pod \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") "
Nov 03 02:41:30.521590 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.521544    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-podinfo\") pod \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") "
Nov 03 02:41:30.521590 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.521580    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istiod-ca-cert\") pod \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") "
Nov 03 02:41:30.521802 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.521611    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-token\" (UniqueName: \"kubernetes.io/projected/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-token\") pod \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") "
Nov 03 02:41:30.521802 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.521657    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-credential-socket" pod "ecdc91da-4a69-4e9a-b46c-12d9cf909aa2" (UID: "ecdc91da-4a69-4e9a-b46c-12d9cf909aa2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:30.521802 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.521644    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-data\") pod \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") "
Nov 03 02:41:30.521802 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.521773    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-envoy\") pod \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") "
Nov 03 02:41:30.522019 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.521832    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-workload-certs" pod "ecdc91da-4a69-4e9a-b46c-12d9cf909aa2" (UID: "ecdc91da-4a69-4e9a-b46c-12d9cf909aa2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:30.522019 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.521918    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-kube-api-access-hblxw\" (UniqueName: \"kubernetes.io/projected/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-kube-api-access-hblxw\") pod \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") "
Nov 03 02:41:30.522019 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.522002    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istiod-ca-cert" pod "ecdc91da-4a69-4e9a-b46c-12d9cf909aa2" (UID: "ecdc91da-4a69-4e9a-b46c-12d9cf909aa2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:41:30.522182 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.521996    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-workload-socket\") pod \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\" (UID: \"ecdc91da-4a69-4e9a-b46c-12d9cf909aa2\") "
Nov 03 02:41:30.522241 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.522202    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-data" pod "ecdc91da-4a69-4e9a-b46c-12d9cf909aa2" (UID: "ecdc91da-4a69-4e9a-b46c-12d9cf909aa2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:30.522323 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.522299    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-workload-socket" pod "ecdc91da-4a69-4e9a-b46c-12d9cf909aa2" (UID: "ecdc91da-4a69-4e9a-b46c-12d9cf909aa2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:30.522730 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.522683    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:30.522730 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.522725    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:30.522881 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.522742    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:30.522881 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.522755    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:30.522881 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.522767    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:30.524264 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.524233    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-podinfo" pod "ecdc91da-4a69-4e9a-b46c-12d9cf909aa2" (UID: "ecdc91da-4a69-4e9a-b46c-12d9cf909aa2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:41:30.524264 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.524233    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-kube-api-access-hblxw" pod "ecdc91da-4a69-4e9a-b46c-12d9cf909aa2" (UID: "ecdc91da-4a69-4e9a-b46c-12d9cf909aa2"). InnerVolumeSpecName "kube-api-access-hblxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:41:30.524453 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.524432    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-envoy" pod "ecdc91da-4a69-4e9a-b46c-12d9cf909aa2" (UID: "ecdc91da-4a69-4e9a-b46c-12d9cf909aa2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:30.524520 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.524511    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-token" pod "ecdc91da-4a69-4e9a-b46c-12d9cf909aa2" (UID: "ecdc91da-4a69-4e9a-b46c-12d9cf909aa2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:41:30.623943 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.623907    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:30.623943 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.623943    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hblxw\" (UniqueName: \"kubernetes.io/projected/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-kube-api-access-hblxw\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:30.624153 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.623958    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:30.624153 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.623970    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:30.717911 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:41:30.717824    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:41:30.897832 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.897790    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"]
Nov 03 02:41:30.945424 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:30.945384    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-mbpwr"]
Nov 03 02:41:31.110732 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:31.110679    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecdc91da-4a69-4e9a-b46c-12d9cf909aa2" path="/var/lib/kubelet/pods/ecdc91da-4a69-4e9a-b46c-12d9cf909aa2/volumes"
Nov 03 02:41:47.637217 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:47.637178    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"]
Nov 03 02:41:47.637780 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:47.637428    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg" podUID="da7afda4-1bc9-418d-8a57-197aae00cb9e" containerName="istio-proxy" containerID="cri-o://00899eb9b54d882e75fe598b93275b93413c34c56a8fec318ef63ef80eb0d523" gracePeriod=30
Nov 03 02:41:52.909270 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.909204    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:41:52.924800 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.924767    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da7afda4-1bc9-418d-8a57-197aae00cb9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da7afda4-1bc9-418d-8a57-197aae00cb9e-istiod-ca-cert\") pod \"da7afda4-1bc9-418d-8a57-197aae00cb9e\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") "
Nov 03 02:41:52.924939 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.924836    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-envoy\") pod \"da7afda4-1bc9-418d-8a57-197aae00cb9e\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") "
Nov 03 02:41:52.924981 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.924933    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-data\") pod \"da7afda4-1bc9-418d-8a57-197aae00cb9e\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") "
Nov 03 02:41:52.925024 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.924995    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-token\" (UniqueName: \"kubernetes.io/projected/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-token\") pod \"da7afda4-1bc9-418d-8a57-197aae00cb9e\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") "
Nov 03 02:41:52.925121 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.925104    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-podinfo\") pod \"da7afda4-1bc9-418d-8a57-197aae00cb9e\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") "
Nov 03 02:41:52.925189 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.925149    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-workload-socket\") pod \"da7afda4-1bc9-418d-8a57-197aae00cb9e\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") "
Nov 03 02:41:52.925247 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.925183    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-data" pod "da7afda4-1bc9-418d-8a57-197aae00cb9e" (UID: "da7afda4-1bc9-418d-8a57-197aae00cb9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:52.925247 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.925191    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7afda4-1bc9-418d-8a57-197aae00cb9e-istiod-ca-cert" pod "da7afda4-1bc9-418d-8a57-197aae00cb9e" (UID: "da7afda4-1bc9-418d-8a57-197aae00cb9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:41:52.925247 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.925209    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-workload-certs\") pod \"da7afda4-1bc9-418d-8a57-197aae00cb9e\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") "
Nov 03 02:41:52.925247 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.925238    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da7afda4-1bc9-418d-8a57-197aae00cb9e-kube-api-access-7kbw8\" (UniqueName: \"kubernetes.io/projected/da7afda4-1bc9-418d-8a57-197aae00cb9e-kube-api-access-7kbw8\") pod \"da7afda4-1bc9-418d-8a57-197aae00cb9e\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") "
Nov 03 02:41:52.925466 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.925382    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-credential-socket\") pod \"da7afda4-1bc9-418d-8a57-197aae00cb9e\" (UID: \"da7afda4-1bc9-418d-8a57-197aae00cb9e\") "
Nov 03 02:41:52.925466 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.925418    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-workload-socket" pod "da7afda4-1bc9-418d-8a57-197aae00cb9e" (UID: "da7afda4-1bc9-418d-8a57-197aae00cb9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:52.925466 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.925437    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-workload-certs" pod "da7afda4-1bc9-418d-8a57-197aae00cb9e" (UID: "da7afda4-1bc9-418d-8a57-197aae00cb9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:52.925737 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.925687    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-credential-socket" pod "da7afda4-1bc9-418d-8a57-197aae00cb9e" (UID: "da7afda4-1bc9-418d-8a57-197aae00cb9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:52.925946 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.925921    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:52.926080 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.925948    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da7afda4-1bc9-418d-8a57-197aae00cb9e-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:52.926080 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.925966    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:52.926080 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.925981    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:52.926080 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.925998    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:52.927067 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.927041    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-envoy" pod "da7afda4-1bc9-418d-8a57-197aae00cb9e" (UID: "da7afda4-1bc9-418d-8a57-197aae00cb9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:52.927267 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.927248    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-token" pod "da7afda4-1bc9-418d-8a57-197aae00cb9e" (UID: "da7afda4-1bc9-418d-8a57-197aae00cb9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:41:52.927323 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.927268    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-podinfo" pod "da7afda4-1bc9-418d-8a57-197aae00cb9e" (UID: "da7afda4-1bc9-418d-8a57-197aae00cb9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:41:52.927426 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:52.927412    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7afda4-1bc9-418d-8a57-197aae00cb9e-kube-api-access-7kbw8" pod "da7afda4-1bc9-418d-8a57-197aae00cb9e" (UID: "da7afda4-1bc9-418d-8a57-197aae00cb9e"). InnerVolumeSpecName "kube-api-access-7kbw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:41:53.027757 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:53.027688    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:53.027757 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:53.027753    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:53.027757 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:53.027764    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da7afda4-1bc9-418d-8a57-197aae00cb9e-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:53.028008 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:53.027775    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kbw8\" (UniqueName: \"kubernetes.io/projected/da7afda4-1bc9-418d-8a57-197aae00cb9e-kube-api-access-7kbw8\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:53.644950 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:53.644914    5005 generic.go:356] "Generic (PLEG): container finished" podID="da7afda4-1bc9-418d-8a57-197aae00cb9e" containerID="00899eb9b54d882e75fe598b93275b93413c34c56a8fec318ef63ef80eb0d523" exitCode=0
Nov 03 02:41:53.645146 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:53.644974    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"
Nov 03 02:41:53.645146 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:53.644997    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg" event={"ID":"da7afda4-1bc9-418d-8a57-197aae00cb9e","Type":"ContainerDied","Data":"00899eb9b54d882e75fe598b93275b93413c34c56a8fec318ef63ef80eb0d523"}
Nov 03 02:41:53.645146 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:53.645052    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg" event={"ID":"da7afda4-1bc9-418d-8a57-197aae00cb9e","Type":"ContainerDied","Data":"f55529e0dc119e1404617f39dd3055b8d3506fc238993df1187f83cdf7508bec"}
Nov 03 02:41:53.645146 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:53.645076    5005 scope.go:122] "RemoveContainer" containerID="00899eb9b54d882e75fe598b93275b93413c34c56a8fec318ef63ef80eb0d523"
Nov 03 02:41:53.655735 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:53.655718    5005 scope.go:122] "RemoveContainer" containerID="00899eb9b54d882e75fe598b93275b93413c34c56a8fec318ef63ef80eb0d523"
Nov 03 02:41:53.656044 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:41:53.656021    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00899eb9b54d882e75fe598b93275b93413c34c56a8fec318ef63ef80eb0d523\": container with ID starting with 00899eb9b54d882e75fe598b93275b93413c34c56a8fec318ef63ef80eb0d523 not found: ID does not exist" containerID="00899eb9b54d882e75fe598b93275b93413c34c56a8fec318ef63ef80eb0d523"
Nov 03 02:41:53.656101 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:53.656054    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00899eb9b54d882e75fe598b93275b93413c34c56a8fec318ef63ef80eb0d523"} err="failed to get container status \"00899eb9b54d882e75fe598b93275b93413c34c56a8fec318ef63ef80eb0d523\": rpc error: code = NotFound desc = could not find container \"00899eb9b54d882e75fe598b93275b93413c34c56a8fec318ef63ef80eb0d523\": container with ID starting with 00899eb9b54d882e75fe598b93275b93413c34c56a8fec318ef63ef80eb0d523 not found: ID does not exist"
Nov 03 02:41:53.800016 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:53.799981    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"]
Nov 03 02:41:53.847350 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:53.847314    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a0e72e-openshift-default-5b4cb49f5f-6pdrg"]
Nov 03 02:41:55.110242 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:41:55.110208    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da7afda4-1bc9-418d-8a57-197aae00cb9e" path="/var/lib/kubelet/pods/da7afda4-1bc9-418d-8a57-197aae00cb9e/volumes"
Nov 03 02:42:00.715285 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:42:00.715251    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:42:30.717780 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:42:30.717741    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:43:00.713777 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:43:00.713687    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:43:30.715324 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:43:30.715285    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:43:46.476203 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:46.476169    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"]
Nov 03 02:43:46.476760 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:46.476361    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4" podUID="bfa108bd-7651-4fe9-9e46-3d9ef40139fe" containerName="istio-proxy" containerID="cri-o://5d5081af51cba7933c980912d182143a0561b8402411cf24282dcafdbb16f915" gracePeriod=30
Nov 03 02:43:51.757042 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.757014    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:43:51.892474 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.892430    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istiod-ca-cert\") pod \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") "
Nov 03 02:43:51.892731 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.892508    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-credential-socket\") pod \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") "
Nov 03 02:43:51.892731 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.892537    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-data\") pod \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") "
Nov 03 02:43:51.892731 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.892559    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-podinfo\") pod \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") "
Nov 03 02:43:51.892731 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.892616    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-workload-certs\") pod \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") "
Nov 03 02:43:51.893775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.892786    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-workload-socket\") pod \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") "
Nov 03 02:43:51.893775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.892839    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-envoy\") pod \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") "
Nov 03 02:43:51.893775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.892839    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-credential-socket" pod "bfa108bd-7651-4fe9-9e46-3d9ef40139fe" (UID: "bfa108bd-7651-4fe9-9e46-3d9ef40139fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:43:51.893775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.892874    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-kube-api-access-ghn27\" (UniqueName: \"kubernetes.io/projected/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-kube-api-access-ghn27\") pod \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") "
Nov 03 02:43:51.893775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.892957    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-token\" (UniqueName: \"kubernetes.io/projected/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-token\") pod \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\" (UID: \"bfa108bd-7651-4fe9-9e46-3d9ef40139fe\") "
Nov 03 02:43:51.893775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.892963    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istiod-ca-cert" pod "bfa108bd-7651-4fe9-9e46-3d9ef40139fe" (UID: "bfa108bd-7651-4fe9-9e46-3d9ef40139fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:43:51.893775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.892944    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-data" pod "bfa108bd-7651-4fe9-9e46-3d9ef40139fe" (UID: "bfa108bd-7651-4fe9-9e46-3d9ef40139fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:43:51.893775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.893017    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-workload-certs" pod "bfa108bd-7651-4fe9-9e46-3d9ef40139fe" (UID: "bfa108bd-7651-4fe9-9e46-3d9ef40139fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:43:51.893775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.893264    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-workload-socket" pod "bfa108bd-7651-4fe9-9e46-3d9ef40139fe" (UID: "bfa108bd-7651-4fe9-9e46-3d9ef40139fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:43:51.893775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.893562    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:43:51.893775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.893585    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:43:51.893775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.893598    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:43:51.893775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.893608    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:43:51.893775 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.893619    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:43:51.895632 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.895608    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-podinfo" pod "bfa108bd-7651-4fe9-9e46-3d9ef40139fe" (UID: "bfa108bd-7651-4fe9-9e46-3d9ef40139fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:43:51.895789 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.895751    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-envoy" pod "bfa108bd-7651-4fe9-9e46-3d9ef40139fe" (UID: "bfa108bd-7651-4fe9-9e46-3d9ef40139fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:43:51.895789 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.895774    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-token" pod "bfa108bd-7651-4fe9-9e46-3d9ef40139fe" (UID: "bfa108bd-7651-4fe9-9e46-3d9ef40139fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:43:51.896118 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.896091    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-kube-api-access-ghn27" pod "bfa108bd-7651-4fe9-9e46-3d9ef40139fe" (UID: "bfa108bd-7651-4fe9-9e46-3d9ef40139fe"). InnerVolumeSpecName "kube-api-access-ghn27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:43:51.995222 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.995176    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:43:51.995222 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.995214    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghn27\" (UniqueName: \"kubernetes.io/projected/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-kube-api-access-ghn27\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:43:51.995222 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.995227    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:43:51.995482 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:51.995241    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa108bd-7651-4fe9-9e46-3d9ef40139fe-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:43:52.456044 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:52.456005    5005 generic.go:356] "Generic (PLEG): container finished" podID="bfa108bd-7651-4fe9-9e46-3d9ef40139fe" containerID="5d5081af51cba7933c980912d182143a0561b8402411cf24282dcafdbb16f915" exitCode=0
Nov 03 02:43:52.456234 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:52.456054    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"
Nov 03 02:43:52.456234 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:52.456089    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4" event={"ID":"bfa108bd-7651-4fe9-9e46-3d9ef40139fe","Type":"ContainerDied","Data":"5d5081af51cba7933c980912d182143a0561b8402411cf24282dcafdbb16f915"}
Nov 03 02:43:52.456234 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:52.456148    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4" event={"ID":"bfa108bd-7651-4fe9-9e46-3d9ef40139fe","Type":"ContainerDied","Data":"b238b42ec7414636ecdc1ca93b7e6c8c07fd58d1cf3e1977d79bfcd18b4876b1"}
Nov 03 02:43:52.456234 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:52.456176    5005 scope.go:122] "RemoveContainer" containerID="5d5081af51cba7933c980912d182143a0561b8402411cf24282dcafdbb16f915"
Nov 03 02:43:52.467865 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:52.467843    5005 scope.go:122] "RemoveContainer" containerID="5d5081af51cba7933c980912d182143a0561b8402411cf24282dcafdbb16f915"
Nov 03 02:43:52.468155 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:43:52.468132    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d5081af51cba7933c980912d182143a0561b8402411cf24282dcafdbb16f915\": container with ID starting with 5d5081af51cba7933c980912d182143a0561b8402411cf24282dcafdbb16f915 not found: ID does not exist" containerID="5d5081af51cba7933c980912d182143a0561b8402411cf24282dcafdbb16f915"
Nov 03 02:43:52.468198 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:52.468167    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d5081af51cba7933c980912d182143a0561b8402411cf24282dcafdbb16f915"} err="failed to get container status \"5d5081af51cba7933c980912d182143a0561b8402411cf24282dcafdbb16f915\": rpc error: code = NotFound desc = could not find container \"5d5081af51cba7933c980912d182143a0561b8402411cf24282dcafdbb16f915\": container with ID starting with 5d5081af51cba7933c980912d182143a0561b8402411cf24282dcafdbb16f915 not found: ID does not exist"
Nov 03 02:43:52.613857 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:52.613818    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"]
Nov 03 02:43:52.661052 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:52.661015    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1bccbd-openshift-default-65f5d49d9-fb6r4"]
Nov 03 02:43:53.110319 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:43:53.110283    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfa108bd-7651-4fe9-9e46-3d9ef40139fe" path="/var/lib/kubelet/pods/bfa108bd-7651-4fe9-9e46-3d9ef40139fe/volumes"
Nov 03 02:44:00.714178 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:44:00.714140    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:44:11.327046 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:11.327012    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"]
Nov 03 02:44:11.327525 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:11.327215    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2" podUID="cefe8bab-136b-428e-8173-c6c89259fe5f" containerName="istio-proxy" containerID="cri-o://5db5d66bfc9ec7d472e85820b52b3184a21300f4ff07ce393f7514c9f5696e49" gracePeriod=30
Nov 03 02:44:14.136266 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:14.136225    5005 patch_prober.go:29] interesting pod/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 02:44:14.136759 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:14.136306    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2" podUID="cefe8bab-136b-428e-8173-c6c89259fe5f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 02:44:16.616776 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.616681    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:44:16.618381 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.618354    5005 generic.go:356] "Generic (PLEG): container finished" podID="cefe8bab-136b-428e-8173-c6c89259fe5f" containerID="5db5d66bfc9ec7d472e85820b52b3184a21300f4ff07ce393f7514c9f5696e49" exitCode=0
Nov 03 02:44:16.618504 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.618396    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"
Nov 03 02:44:16.618504 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.618437    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2" event={"ID":"cefe8bab-136b-428e-8173-c6c89259fe5f","Type":"ContainerDied","Data":"5db5d66bfc9ec7d472e85820b52b3184a21300f4ff07ce393f7514c9f5696e49"}
Nov 03 02:44:16.618598 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.618524    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2" event={"ID":"cefe8bab-136b-428e-8173-c6c89259fe5f","Type":"ContainerDied","Data":"71b1e1045b0c2ce703bb513d3499aeb0163cb02b3177f72f74c319e9845730e3"}
Nov 03 02:44:16.618598 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.618551    5005 scope.go:122] "RemoveContainer" containerID="5db5d66bfc9ec7d472e85820b52b3184a21300f4ff07ce393f7514c9f5696e49"
Nov 03 02:44:16.628759 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.628737    5005 scope.go:122] "RemoveContainer" containerID="5db5d66bfc9ec7d472e85820b52b3184a21300f4ff07ce393f7514c9f5696e49"
Nov 03 02:44:16.629028 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:44:16.629008    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5db5d66bfc9ec7d472e85820b52b3184a21300f4ff07ce393f7514c9f5696e49\": container with ID starting with 5db5d66bfc9ec7d472e85820b52b3184a21300f4ff07ce393f7514c9f5696e49 not found: ID does not exist" containerID="5db5d66bfc9ec7d472e85820b52b3184a21300f4ff07ce393f7514c9f5696e49"
Nov 03 02:44:16.629098 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.629034    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5db5d66bfc9ec7d472e85820b52b3184a21300f4ff07ce393f7514c9f5696e49"} err="failed to get container status \"5db5d66bfc9ec7d472e85820b52b3184a21300f4ff07ce393f7514c9f5696e49\": rpc error: code = NotFound desc = could not find container \"5db5d66bfc9ec7d472e85820b52b3184a21300f4ff07ce393f7514c9f5696e49\": container with ID starting with 5db5d66bfc9ec7d472e85820b52b3184a21300f4ff07ce393f7514c9f5696e49 not found: ID does not exist"
Nov 03 02:44:16.702604 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.702569    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-credential-socket\") pod \"cefe8bab-136b-428e-8173-c6c89259fe5f\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") "
Nov 03 02:44:16.702822 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.702630    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cefe8bab-136b-428e-8173-c6c89259fe5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefe8bab-136b-428e-8173-c6c89259fe5f-istiod-ca-cert\") pod \"cefe8bab-136b-428e-8173-c6c89259fe5f\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") "
Nov 03 02:44:16.702822 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.702672    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-podinfo\") pod \"cefe8bab-136b-428e-8173-c6c89259fe5f\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") "
Nov 03 02:44:16.702822 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.702737    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-data\") pod \"cefe8bab-136b-428e-8173-c6c89259fe5f\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") "
Nov 03 02:44:16.703000 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.702869    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-token\" (UniqueName: \"kubernetes.io/projected/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-token\") pod \"cefe8bab-136b-428e-8173-c6c89259fe5f\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") "
Nov 03 02:44:16.703000 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.702902    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cefe8bab-136b-428e-8173-c6c89259fe5f-kube-api-access-hft4b\" (UniqueName: \"kubernetes.io/projected/cefe8bab-136b-428e-8173-c6c89259fe5f-kube-api-access-hft4b\") pod \"cefe8bab-136b-428e-8173-c6c89259fe5f\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") "
Nov 03 02:44:16.703000 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.702973    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-envoy\") pod \"cefe8bab-136b-428e-8173-c6c89259fe5f\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") "
Nov 03 02:44:16.703170 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.703063    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-workload-socket\") pod \"cefe8bab-136b-428e-8173-c6c89259fe5f\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") "
Nov 03 02:44:16.703170 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.703129    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-workload-certs\") pod \"cefe8bab-136b-428e-8173-c6c89259fe5f\" (UID: \"cefe8bab-136b-428e-8173-c6c89259fe5f\") "
Nov 03 02:44:16.703271 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.702902    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-credential-socket" pod "cefe8bab-136b-428e-8173-c6c89259fe5f" (UID: "cefe8bab-136b-428e-8173-c6c89259fe5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:44:16.703271 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.703183    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cefe8bab-136b-428e-8173-c6c89259fe5f-istiod-ca-cert" pod "cefe8bab-136b-428e-8173-c6c89259fe5f" (UID: "cefe8bab-136b-428e-8173-c6c89259fe5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:44:16.703271 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.703195    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-data" pod "cefe8bab-136b-428e-8173-c6c89259fe5f" (UID: "cefe8bab-136b-428e-8173-c6c89259fe5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:44:16.703491 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.703466    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-workload-certs" pod "cefe8bab-136b-428e-8173-c6c89259fe5f" (UID: "cefe8bab-136b-428e-8173-c6c89259fe5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:44:16.703568 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.703478    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-workload-socket" pod "cefe8bab-136b-428e-8173-c6c89259fe5f" (UID: "cefe8bab-136b-428e-8173-c6c89259fe5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:44:16.703768 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.703743    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:16.703958 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.703941    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefe8bab-136b-428e-8173-c6c89259fe5f-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:16.704085 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.704060    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:16.704085 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.704085    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:16.704257 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.704101    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:16.705358 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.705335    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-envoy" pod "cefe8bab-136b-428e-8173-c6c89259fe5f" (UID: "cefe8bab-136b-428e-8173-c6c89259fe5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:44:16.705524 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.705500    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-token" pod "cefe8bab-136b-428e-8173-c6c89259fe5f" (UID: "cefe8bab-136b-428e-8173-c6c89259fe5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:44:16.705777 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.705755    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cefe8bab-136b-428e-8173-c6c89259fe5f-kube-api-access-hft4b" pod "cefe8bab-136b-428e-8173-c6c89259fe5f" (UID: "cefe8bab-136b-428e-8173-c6c89259fe5f"). InnerVolumeSpecName "kube-api-access-hft4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:44:16.705850 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.705759    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-podinfo" pod "cefe8bab-136b-428e-8173-c6c89259fe5f" (UID: "cefe8bab-136b-428e-8173-c6c89259fe5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:44:16.805804 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.805767    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:16.805804 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.805801    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hft4b\" (UniqueName: \"kubernetes.io/projected/cefe8bab-136b-428e-8173-c6c89259fe5f-kube-api-access-hft4b\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:16.805804 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.805813    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:16.806048 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:16.805821    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefe8bab-136b-428e-8173-c6c89259fe5f-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:17.035786 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:17.035554    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"]
Nov 03 02:44:17.039839 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:17.039804    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f7d9bb-openshift-default-79cbf96dd9-km5s2"]
Nov 03 02:44:17.110908 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:17.110872    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cefe8bab-136b-428e-8173-c6c89259fe5f" path="/var/lib/kubelet/pods/cefe8bab-136b-428e-8173-c6c89259fe5f/volumes"
Nov 03 02:44:25.745642 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:25.745553    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"]
Nov 03 02:44:25.746175 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:25.745807    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t" podUID="ee116929-8ee5-40c2-9384-4fe02d40c720" containerName="maas-api" containerID="cri-o://f57416d93fb00f71c58361e82737569c2be035bc8984d6118a9f35b8468f3b1d" gracePeriod=30
Nov 03 02:44:26.044655 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:26.044625    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"
Nov 03 02:44:26.153913 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:26.152501    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee116929-8ee5-40c2-9384-4fe02d40c720-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee116929-8ee5-40c2-9384-4fe02d40c720-metrics-certs\") pod \"ee116929-8ee5-40c2-9384-4fe02d40c720\" (UID: \"ee116929-8ee5-40c2-9384-4fe02d40c720\") "
Nov 03 02:44:26.153913 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:26.152592    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee116929-8ee5-40c2-9384-4fe02d40c720-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee116929-8ee5-40c2-9384-4fe02d40c720-openshift-service-ca\") pod \"ee116929-8ee5-40c2-9384-4fe02d40c720\" (UID: \"ee116929-8ee5-40c2-9384-4fe02d40c720\") "
Nov 03 02:44:26.153913 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:26.152741    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee116929-8ee5-40c2-9384-4fe02d40c720-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee116929-8ee5-40c2-9384-4fe02d40c720-maas-api-tls\") pod \"ee116929-8ee5-40c2-9384-4fe02d40c720\" (UID: \"ee116929-8ee5-40c2-9384-4fe02d40c720\") "
Nov 03 02:44:26.153913 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:26.152819    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee116929-8ee5-40c2-9384-4fe02d40c720-kube-api-access-nlhjl\" (UniqueName: \"kubernetes.io/projected/ee116929-8ee5-40c2-9384-4fe02d40c720-kube-api-access-nlhjl\") pod \"ee116929-8ee5-40c2-9384-4fe02d40c720\" (UID: \"ee116929-8ee5-40c2-9384-4fe02d40c720\") "
Nov 03 02:44:26.154925 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:26.154881    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee116929-8ee5-40c2-9384-4fe02d40c720-openshift-service-ca" pod "ee116929-8ee5-40c2-9384-4fe02d40c720" (UID: "ee116929-8ee5-40c2-9384-4fe02d40c720"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:44:26.174499 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:26.171127    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee116929-8ee5-40c2-9384-4fe02d40c720-kube-api-access-nlhjl" pod "ee116929-8ee5-40c2-9384-4fe02d40c720" (UID: "ee116929-8ee5-40c2-9384-4fe02d40c720"). InnerVolumeSpecName "kube-api-access-nlhjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:44:26.174499 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:26.171795    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee116929-8ee5-40c2-9384-4fe02d40c720-maas-api-tls" pod "ee116929-8ee5-40c2-9384-4fe02d40c720" (UID: "ee116929-8ee5-40c2-9384-4fe02d40c720"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:44:26.174499 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:26.173109    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee116929-8ee5-40c2-9384-4fe02d40c720-metrics-certs" pod "ee116929-8ee5-40c2-9384-4fe02d40c720" (UID: "ee116929-8ee5-40c2-9384-4fe02d40c720"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:44:26.254362 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:26.254319    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee116929-8ee5-40c2-9384-4fe02d40c720-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:26.254362 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:26.254357    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee116929-8ee5-40c2-9384-4fe02d40c720-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:26.254362 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:26.254372    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee116929-8ee5-40c2-9384-4fe02d40c720-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:26.254610 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:26.254387    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlhjl\" (UniqueName: \"kubernetes.io/projected/ee116929-8ee5-40c2-9384-4fe02d40c720-kube-api-access-nlhjl\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:26.728717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:26.727423    5005 generic.go:356] "Generic (PLEG): container finished" podID="ee116929-8ee5-40c2-9384-4fe02d40c720" containerID="f57416d93fb00f71c58361e82737569c2be035bc8984d6118a9f35b8468f3b1d" exitCode=0
Nov 03 02:44:26.728717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:26.727659    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t" event={"ID":"ee116929-8ee5-40c2-9384-4fe02d40c720","Type":"ContainerDied","Data":"f57416d93fb00f71c58361e82737569c2be035bc8984d6118a9f35b8468f3b1d"}
Nov 03 02:44:26.728717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:26.727741    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t" event={"ID":"ee116929-8ee5-40c2-9384-4fe02d40c720","Type":"ContainerDied","Data":"c3f31783864fee3442a0cb665f2dcbf3493c1921f2ef4ad05f91ab6e0cb49317"}
Nov 03 02:44:26.728717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:26.727775    5005 scope.go:122] "RemoveContainer" containerID="f57416d93fb00f71c58361e82737569c2be035bc8984d6118a9f35b8468f3b1d"
Nov 03 02:44:26.728717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:26.727949    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"
Nov 03 02:44:26.753743 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:26.753686    5005 scope.go:122] "RemoveContainer" containerID="f57416d93fb00f71c58361e82737569c2be035bc8984d6118a9f35b8468f3b1d"
Nov 03 02:44:26.754555 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:44:26.754527    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57416d93fb00f71c58361e82737569c2be035bc8984d6118a9f35b8468f3b1d\": container with ID starting with f57416d93fb00f71c58361e82737569c2be035bc8984d6118a9f35b8468f3b1d not found: ID does not exist" containerID="f57416d93fb00f71c58361e82737569c2be035bc8984d6118a9f35b8468f3b1d"
Nov 03 02:44:26.754665 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:26.754568    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57416d93fb00f71c58361e82737569c2be035bc8984d6118a9f35b8468f3b1d"} err="failed to get container status \"f57416d93fb00f71c58361e82737569c2be035bc8984d6118a9f35b8468f3b1d\": rpc error: code = NotFound desc = could not find container \"f57416d93fb00f71c58361e82737569c2be035bc8984d6118a9f35b8468f3b1d\": container with ID starting with f57416d93fb00f71c58361e82737569c2be035bc8984d6118a9f35b8468f3b1d not found: ID does not exist"
Nov 03 02:44:26.885089 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:26.885048    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"]
Nov 03 02:44:26.930645 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:26.930609    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f954db-754986b779-j9k2t"]
Nov 03 02:44:27.109710 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:27.109660    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee116929-8ee5-40c2-9384-4fe02d40c720" path="/var/lib/kubelet/pods/ee116929-8ee5-40c2-9384-4fe02d40c720/volumes"
Nov 03 02:44:29.607161 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:29.607122    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"]
Nov 03 02:44:29.607631 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:29.607324    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk" podUID="1fe7c918-b791-4dd9-bb28-85c79fc9b8c7" containerName="istio-proxy" containerID="cri-o://2654d77c32f27aea72020987d22b94c234513157f5672e20718d4c847b09ef9d" gracePeriod=30
Nov 03 02:44:30.714975 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:44:30.714935    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:44:33.172707 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:33.172655    5005 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 02:44:33.173122 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:33.172735    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk" podUID="1fe7c918-b791-4dd9-bb28-85c79fc9b8c7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 02:44:34.810247 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.810207    5005 generic.go:356] "Generic (PLEG): container finished" podID="1fe7c918-b791-4dd9-bb28-85c79fc9b8c7" containerID="2654d77c32f27aea72020987d22b94c234513157f5672e20718d4c847b09ef9d" exitCode=0
Nov 03 02:44:34.810653 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.810270    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk" event={"ID":"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7","Type":"ContainerDied","Data":"2654d77c32f27aea72020987d22b94c234513157f5672e20718d4c847b09ef9d"}
Nov 03 02:44:34.893903 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.893874    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:44:34.985476 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.985432    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-envoy\") pod \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") "
Nov 03 02:44:34.985476 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.985476    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-kube-api-access-l6gmd\" (UniqueName: \"kubernetes.io/projected/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-kube-api-access-l6gmd\") pod \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") "
Nov 03 02:44:34.985753 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.985524    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-workload-socket\") pod \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") "
Nov 03 02:44:34.985753 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.985556    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-data\") pod \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") "
Nov 03 02:44:34.985753 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.985590    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-podinfo\") pod \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") "
Nov 03 02:44:34.985753 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.985683    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-workload-certs\") pod \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") "
Nov 03 02:44:34.985753 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.985752    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istiod-ca-cert\") pod \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") "
Nov 03 02:44:34.986033 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.985840    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-token\" (UniqueName: \"kubernetes.io/projected/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-token\") pod \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") "
Nov 03 02:44:34.986033 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.985871    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-credential-socket\") pod \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\" (UID: \"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7\") "
Nov 03 02:44:34.986033 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.985902    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-workload-socket" pod "1fe7c918-b791-4dd9-bb28-85c79fc9b8c7" (UID: "1fe7c918-b791-4dd9-bb28-85c79fc9b8c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:44:34.986033 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.985957    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-data" pod "1fe7c918-b791-4dd9-bb28-85c79fc9b8c7" (UID: "1fe7c918-b791-4dd9-bb28-85c79fc9b8c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:44:34.986232 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.986178    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istiod-ca-cert" pod "1fe7c918-b791-4dd9-bb28-85c79fc9b8c7" (UID: "1fe7c918-b791-4dd9-bb28-85c79fc9b8c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:44:34.986295 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.986226    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-credential-socket" pod "1fe7c918-b791-4dd9-bb28-85c79fc9b8c7" (UID: "1fe7c918-b791-4dd9-bb28-85c79fc9b8c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:44:34.986350 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.986290    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-workload-certs" pod "1fe7c918-b791-4dd9-bb28-85c79fc9b8c7" (UID: "1fe7c918-b791-4dd9-bb28-85c79fc9b8c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:44:34.986447 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.986427    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:34.986508 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.986448    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:34.986508 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.986462    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:34.986508 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.986475    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:34.986508 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.986490    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:34.987986 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.987965    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-envoy" pod "1fe7c918-b791-4dd9-bb28-85c79fc9b8c7" (UID: "1fe7c918-b791-4dd9-bb28-85c79fc9b8c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:44:34.988087 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.988056    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-token" pod "1fe7c918-b791-4dd9-bb28-85c79fc9b8c7" (UID: "1fe7c918-b791-4dd9-bb28-85c79fc9b8c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:44:34.988201 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.988185    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-podinfo" pod "1fe7c918-b791-4dd9-bb28-85c79fc9b8c7" (UID: "1fe7c918-b791-4dd9-bb28-85c79fc9b8c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:44:34.988334 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:34.988319    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-kube-api-access-l6gmd" pod "1fe7c918-b791-4dd9-bb28-85c79fc9b8c7" (UID: "1fe7c918-b791-4dd9-bb28-85c79fc9b8c7"). InnerVolumeSpecName "kube-api-access-l6gmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:44:35.087809 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:35.087769    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:35.087809 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:35.087808    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:35.088031 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:35.087823    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:35.088031 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:35.087838    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6gmd\" (UniqueName: \"kubernetes.io/projected/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7-kube-api-access-l6gmd\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:35.821162 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:35.821114    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk" event={"ID":"1fe7c918-b791-4dd9-bb28-85c79fc9b8c7","Type":"ContainerDied","Data":"5770683220d96351cba53d11abf20d60768f6361340ed7e6dc8ec70d6692e362"}
Nov 03 02:44:35.821751 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:35.821238    5005 scope.go:122] "RemoveContainer" containerID="2654d77c32f27aea72020987d22b94c234513157f5672e20718d4c847b09ef9d"
Nov 03 02:44:35.821751 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:35.821443    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"
Nov 03 02:44:35.997625 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:35.997587    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"]
Nov 03 02:44:36.044100 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:36.044064    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-cfjbk"]
Nov 03 02:44:37.110155 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.110118    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe7c918-b791-4dd9-bb28-85c79fc9b8c7" path="/var/lib/kubelet/pods/1fe7c918-b791-4dd9-bb28-85c79fc9b8c7/volumes"
Nov 03 02:44:37.150628 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.150594    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"]
Nov 03 02:44:37.150842 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.150812    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th" podUID="f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da" containerName="maas-api" containerID="cri-o://7310319571d8fb5f4167b9b2d22515cac200811c6cf944b7453c197484bcd4d7" gracePeriod=30
Nov 03 02:44:37.461511 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.461479    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"
Nov 03 02:44:37.516857 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.516825    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-maas-api-tls\") pod \"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da\" (UID: \"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da\") "
Nov 03 02:44:37.517070 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.516870    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-kube-api-access-6ql4l\" (UniqueName: \"kubernetes.io/projected/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-kube-api-access-6ql4l\") pod \"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da\" (UID: \"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da\") "
Nov 03 02:44:37.517070 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.516938    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-openshift-service-ca\") pod \"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da\" (UID: \"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da\") "
Nov 03 02:44:37.517070 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.516961    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-metrics-certs\") pod \"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da\" (UID: \"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da\") "
Nov 03 02:44:37.517297 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.517274    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-openshift-service-ca" pod "f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da" (UID: "f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:44:37.517434 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.517416    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:37.519251 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.519221    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-maas-api-tls" pod "f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da" (UID: "f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:44:37.519378 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.519281    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-kube-api-access-6ql4l" pod "f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da" (UID: "f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da"). InnerVolumeSpecName "kube-api-access-6ql4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:44:37.519378 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.519299    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-metrics-certs" pod "f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da" (UID: "f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:44:37.619492 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.619443    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:37.619492 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.619488    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ql4l\" (UniqueName: \"kubernetes.io/projected/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-kube-api-access-6ql4l\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:37.619732 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.619505    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:37.852316 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.852277    5005 generic.go:356] "Generic (PLEG): container finished" podID="f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da" containerID="7310319571d8fb5f4167b9b2d22515cac200811c6cf944b7453c197484bcd4d7" exitCode=0
Nov 03 02:44:37.852528 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.852372    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th" event={"ID":"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da","Type":"ContainerDied","Data":"7310319571d8fb5f4167b9b2d22515cac200811c6cf944b7453c197484bcd4d7"}
Nov 03 02:44:37.852528 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.852424    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th" event={"ID":"f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da","Type":"ContainerDied","Data":"887c792ba801e69fcfd5f2ee78216587ae91f5ce3e8b1885bac620b5ae7221e8"}
Nov 03 02:44:37.852528 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.852454    5005 scope.go:122] "RemoveContainer" containerID="7310319571d8fb5f4167b9b2d22515cac200811c6cf944b7453c197484bcd4d7"
Nov 03 02:44:37.852727 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.852618    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"
Nov 03 02:44:37.892129 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.890297    5005 scope.go:122] "RemoveContainer" containerID="7310319571d8fb5f4167b9b2d22515cac200811c6cf944b7453c197484bcd4d7"
Nov 03 02:44:37.892129 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:44:37.891516    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7310319571d8fb5f4167b9b2d22515cac200811c6cf944b7453c197484bcd4d7\": container with ID starting with 7310319571d8fb5f4167b9b2d22515cac200811c6cf944b7453c197484bcd4d7 not found: ID does not exist" containerID="7310319571d8fb5f4167b9b2d22515cac200811c6cf944b7453c197484bcd4d7"
Nov 03 02:44:37.892129 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:37.891579    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7310319571d8fb5f4167b9b2d22515cac200811c6cf944b7453c197484bcd4d7"} err="failed to get container status \"7310319571d8fb5f4167b9b2d22515cac200811c6cf944b7453c197484bcd4d7\": rpc error: code = NotFound desc = could not find container \"7310319571d8fb5f4167b9b2d22515cac200811c6cf944b7453c197484bcd4d7\": container with ID starting with 7310319571d8fb5f4167b9b2d22515cac200811c6cf944b7453c197484bcd4d7 not found: ID does not exist"
Nov 03 02:44:38.013560 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:38.013518    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"]
Nov 03 02:44:38.016749 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:38.016719    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4616ec-757d7f4bdc-fm6th"]
Nov 03 02:44:39.109993 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:44:39.109961    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da" path="/var/lib/kubelet/pods/f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da/volumes"
Nov 03 02:45:00.714201 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:45:00.714167    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:45:30.715350 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:45:30.715306    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:46:00.715865 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:46:00.715782    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:46:13.592830 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:46:13.592802    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:46:13.603809 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:46:13.603787    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:46:30.719004 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:46:30.718966    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:47:00.713828 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:47:00.713789    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:47:30.717350 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:47:30.717258    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:47:35.660081 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660044    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"]
Nov 03 02:47:35.660632 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660618    5005 cpu_manager.go:414] "containerMap: removing container" podUID="1fc5f95b-3ced-444a-aab9-39c666ac1a29" containerName="istio-proxy"
Nov 03 02:47:35.660679 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660633    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fc5f95b-3ced-444a-aab9-39c666ac1a29" containerName="istio-proxy"
Nov 03 02:47:35.660679 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660646    5005 cpu_manager.go:414] "containerMap: removing container" podUID="5b97ff37-eae3-4020-86ed-71477662e225" containerName="istio-proxy"
Nov 03 02:47:35.660679 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660651    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b97ff37-eae3-4020-86ed-71477662e225" containerName="istio-proxy"
Nov 03 02:47:35.660679 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660658    5005 cpu_manager.go:414] "containerMap: removing container" podUID="da7afda4-1bc9-418d-8a57-197aae00cb9e" containerName="istio-proxy"
Nov 03 02:47:35.660679 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660663    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7afda4-1bc9-418d-8a57-197aae00cb9e" containerName="istio-proxy"
Nov 03 02:47:35.660679 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660678    5005 cpu_manager.go:414] "containerMap: removing container" podUID="cefe8bab-136b-428e-8173-c6c89259fe5f" containerName="istio-proxy"
Nov 03 02:47:35.660997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660683    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cefe8bab-136b-428e-8173-c6c89259fe5f" containerName="istio-proxy"
Nov 03 02:47:35.660997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660708    5005 cpu_manager.go:414] "containerMap: removing container" podUID="ecdc91da-4a69-4e9a-b46c-12d9cf909aa2" containerName="istio-proxy"
Nov 03 02:47:35.660997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660715    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecdc91da-4a69-4e9a-b46c-12d9cf909aa2" containerName="istio-proxy"
Nov 03 02:47:35.660997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660739    5005 cpu_manager.go:414] "containerMap: removing container" podUID="ee116929-8ee5-40c2-9384-4fe02d40c720" containerName="maas-api"
Nov 03 02:47:35.660997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660747    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee116929-8ee5-40c2-9384-4fe02d40c720" containerName="maas-api"
Nov 03 02:47:35.660997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660770    5005 cpu_manager.go:414] "containerMap: removing container" podUID="1fe7c918-b791-4dd9-bb28-85c79fc9b8c7" containerName="istio-proxy"
Nov 03 02:47:35.660997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660778    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe7c918-b791-4dd9-bb28-85c79fc9b8c7" containerName="istio-proxy"
Nov 03 02:47:35.660997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660793    5005 cpu_manager.go:414] "containerMap: removing container" podUID="bfa108bd-7651-4fe9-9e46-3d9ef40139fe" containerName="istio-proxy"
Nov 03 02:47:35.660997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660798    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa108bd-7651-4fe9-9e46-3d9ef40139fe" containerName="istio-proxy"
Nov 03 02:47:35.660997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660808    5005 cpu_manager.go:414] "containerMap: removing container" podUID="f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da" containerName="maas-api"
Nov 03 02:47:35.660997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660812    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da" containerName="maas-api"
Nov 03 02:47:35.660997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660920    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe7c918-b791-4dd9-bb28-85c79fc9b8c7" containerName="istio-proxy"
Nov 03 02:47:35.660997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660928    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa108bd-7651-4fe9-9e46-3d9ef40139fe" containerName="istio-proxy"
Nov 03 02:47:35.660997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660936    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="f73f1f0f-6ffa-4ed7-b8cb-b578f56f58da" containerName="maas-api"
Nov 03 02:47:35.660997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660945    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fc5f95b-3ced-444a-aab9-39c666ac1a29" containerName="istio-proxy"
Nov 03 02:47:35.660997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660950    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b97ff37-eae3-4020-86ed-71477662e225" containerName="istio-proxy"
Nov 03 02:47:35.660997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660956    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="da7afda4-1bc9-418d-8a57-197aae00cb9e" containerName="istio-proxy"
Nov 03 02:47:35.660997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660964    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="cefe8bab-136b-428e-8173-c6c89259fe5f" containerName="istio-proxy"
Nov 03 02:47:35.660997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660969    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecdc91da-4a69-4e9a-b46c-12d9cf909aa2" containerName="istio-proxy"
Nov 03 02:47:35.660997 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.660976    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee116929-8ee5-40c2-9384-4fe02d40c720" containerName="maas-api"
Nov 03 02:47:35.664572 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.664555    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"
Nov 03 02:47:35.750438 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.750399    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-metrics-certs\") pod \"maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph\" (UID: \"39eeb830-577b-4fcf-9a4b-9d90c15c8be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"
Nov 03 02:47:35.750628 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.750454    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph\" (UID: \"39eeb830-577b-4fcf-9a4b-9d90c15c8be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"
Nov 03 02:47:35.750681 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.750624    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjbf4\" (UniqueName: \"kubernetes.io/projected/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-kube-api-access-fjbf4\") pod \"maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph\" (UID: \"39eeb830-577b-4fcf-9a4b-9d90c15c8be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"
Nov 03 02:47:35.750681 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.750661    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph\" (UID: \"39eeb830-577b-4fcf-9a4b-9d90c15c8be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"
Nov 03 02:47:35.760687 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.760655    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-ca6ef4-serving-cert\""
Nov 03 02:47:35.764049 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.764026    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"]
Nov 03 02:47:35.851587 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.851551    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjbf4\" (UniqueName: \"kubernetes.io/projected/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-kube-api-access-fjbf4\") pod \"maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph\" (UID: \"39eeb830-577b-4fcf-9a4b-9d90c15c8be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"
Nov 03 02:47:35.851587 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.851591    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph\" (UID: \"39eeb830-577b-4fcf-9a4b-9d90c15c8be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"
Nov 03 02:47:35.851829 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.851718    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-metrics-certs\") pod \"maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph\" (UID: \"39eeb830-577b-4fcf-9a4b-9d90c15c8be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"
Nov 03 02:47:35.851829 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.851750    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph\" (UID: \"39eeb830-577b-4fcf-9a4b-9d90c15c8be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"
Nov 03 02:47:35.851829 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:47:35.851794    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-serving-cert: secret "maas-api-e2e-extmod-a-ca6ef4-serving-cert" not found
Nov 03 02:47:35.851951 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:47:35.851870    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-maas-api-tls podName:39eeb830-577b-4fcf-9a4b-9d90c15c8be4 nodeName:}" failed. No retries permitted until 2026-11-03 02:47:36.351850114 +0000 UTC m=+3383.366091473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-maas-api-tls") pod "maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph" (UID: "39eeb830-577b-4fcf-9a4b-9d90c15c8be4") : secret "maas-api-e2e-extmod-a-ca6ef4-serving-cert" not found
Nov 03 02:47:35.852514 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.852489    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph\" (UID: \"39eeb830-577b-4fcf-9a4b-9d90c15c8be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"
Nov 03 02:47:35.854211 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.854183    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-metrics-certs\") pod \"maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph\" (UID: \"39eeb830-577b-4fcf-9a4b-9d90c15c8be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"
Nov 03 02:47:35.919926 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:35.919847    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjbf4\" (UniqueName: \"kubernetes.io/projected/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-kube-api-access-fjbf4\") pod \"maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph\" (UID: \"39eeb830-577b-4fcf-9a4b-9d90c15c8be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"
Nov 03 02:47:36.356908 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:36.356871    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph\" (UID: \"39eeb830-577b-4fcf-9a4b-9d90c15c8be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"
Nov 03 02:47:36.359448 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:36.359423    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph\" (UID: \"39eeb830-577b-4fcf-9a4b-9d90c15c8be4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"
Nov 03 02:47:36.578809 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:36.578775    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"
Nov 03 02:47:36.955074 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:36.954991    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"]
Nov 03 02:47:36.960586 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:47:36.960559    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39eeb830_577b_4fcf_9a4b_9d90c15c8be4.slice/crio-706f404ae2908deab17821e32567de73e527ca3b175db3975e0e91d421721d3b WatchSource:0}: Error finding container 706f404ae2908deab17821e32567de73e527ca3b175db3975e0e91d421721d3b: Status 404 returned error can't find the container with id 706f404ae2908deab17821e32567de73e527ca3b175db3975e0e91d421721d3b
Nov 03 02:47:36.962410 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:36.962393    5005 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 02:47:37.061754 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:37.061716    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph" event={"ID":"39eeb830-577b-4fcf-9a4b-9d90c15c8be4","Type":"ContainerStarted","Data":"706f404ae2908deab17821e32567de73e527ca3b175db3975e0e91d421721d3b"}
Nov 03 02:47:38.070371 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:38.070338    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph" event={"ID":"39eeb830-577b-4fcf-9a4b-9d90c15c8be4","Type":"ContainerStarted","Data":"0ca278cd34a070be959e9cbe3093f8a0c7509e29f895a22b9fa97c08799fb3bc"}
Nov 03 02:47:38.070842 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:38.070407    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"
Nov 03 02:47:38.179788 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:38.179660    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph" podStartSLOduration=2.2624813169999998 podStartE2EDuration="3.179639972s" totalImagesPullingTime="917.158655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:47:38.177636527 +0000 UTC m=+3385.191877893" watchObservedRunningTime="2026-11-03 02:47:38.179639972 +0000 UTC m=+3385.193881339"
Nov 03 02:47:44.083284 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:44.083251    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"
Nov 03 02:47:49.952366 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:49.952330    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"]
Nov 03 02:47:49.956570 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:49.956548    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"
Nov 03 02:47:50.024749 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:50.024708    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b\" (UID: \"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"
Nov 03 02:47:50.024915 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:50.024822    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pwhz\" (UniqueName: \"kubernetes.io/projected/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-kube-api-access-6pwhz\") pod \"maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b\" (UID: \"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"
Nov 03 02:47:50.024964 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:50.024941    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-maas-api-tls\") pod \"maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b\" (UID: \"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"
Nov 03 02:47:50.025055 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:50.025042    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-metrics-certs\") pod \"maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b\" (UID: \"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"
Nov 03 02:47:50.053030 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:50.052990    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-604da0-serving-cert\""
Nov 03 02:47:50.056503 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:50.056470    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"]
Nov 03 02:47:50.125913 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:50.125873    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-maas-api-tls\") pod \"maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b\" (UID: \"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"
Nov 03 02:47:50.126076 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:47:50.125969    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-serving-cert: secret "maas-api-e2e-extmod-a-604da0-serving-cert" not found
Nov 03 02:47:50.126076 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:50.126001    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-metrics-certs\") pod \"maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b\" (UID: \"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"
Nov 03 02:47:50.126076 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:47:50.126021    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-maas-api-tls podName:df56d3fe-d8fa-40a1-80fc-193ea6ed4a29 nodeName:}" failed. No retries permitted until 2026-11-03 02:47:50.626007348 +0000 UTC m=+3397.640248692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-maas-api-tls") pod "maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b" (UID: "df56d3fe-d8fa-40a1-80fc-193ea6ed4a29") : secret "maas-api-e2e-extmod-a-604da0-serving-cert" not found
Nov 03 02:47:50.126212 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:50.126084    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b\" (UID: \"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"
Nov 03 02:47:50.126212 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:50.126121    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pwhz\" (UniqueName: \"kubernetes.io/projected/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-kube-api-access-6pwhz\") pod \"maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b\" (UID: \"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"
Nov 03 02:47:50.127120 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:50.127086    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b\" (UID: \"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"
Nov 03 02:47:50.128515 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:50.128492    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-metrics-certs\") pod \"maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b\" (UID: \"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"
Nov 03 02:47:50.179477 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:50.179437    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pwhz\" (UniqueName: \"kubernetes.io/projected/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-kube-api-access-6pwhz\") pod \"maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b\" (UID: \"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"
Nov 03 02:47:50.632841 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:50.632801    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-maas-api-tls\") pod \"maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b\" (UID: \"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"
Nov 03 02:47:50.635269 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:50.635234    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-maas-api-tls\") pod \"maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b\" (UID: \"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"
Nov 03 02:47:50.870365 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:50.870332    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"
Nov 03 02:47:51.250981 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:51.250888    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"]
Nov 03 02:47:51.256309 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:47:51.256279    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf56d3fe_d8fa_40a1_80fc_193ea6ed4a29.slice/crio-15963b5df98947b2b0cbd4cd0862c298229cde49d94cd236829bf2ae90e56c7b WatchSource:0}: Error finding container 15963b5df98947b2b0cbd4cd0862c298229cde49d94cd236829bf2ae90e56c7b: Status 404 returned error can't find the container with id 15963b5df98947b2b0cbd4cd0862c298229cde49d94cd236829bf2ae90e56c7b
Nov 03 02:47:52.158386 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:52.158352    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b" event={"ID":"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29","Type":"ContainerStarted","Data":"15963b5df98947b2b0cbd4cd0862c298229cde49d94cd236829bf2ae90e56c7b"}
Nov 03 02:47:53.166138 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:53.166097    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b" event={"ID":"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29","Type":"ContainerStarted","Data":"ad64e9d5ccdac0f8accbcb3c3af8ccbf6a9a6bc0a21c2a6f750048a4bc78ecfd"}
Nov 03 02:47:53.166563 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:53.166171    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"
Nov 03 02:47:53.273492 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:53.273440    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b" podStartSLOduration=3.432413999 podStartE2EDuration="4.273425431s" totalImagesPullingTime="841.011432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:47:53.272529784 +0000 UTC m=+3400.286771151" watchObservedRunningTime="2026-11-03 02:47:53.273425431 +0000 UTC m=+3400.287666805"
Nov 03 02:47:59.178651 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:47:59.178619    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"
Nov 03 02:48:00.715128 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:48:00.715086    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:48:30.714910 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:48:30.714870    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:49:00.715176 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:49:00.715090    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:49:30.714481 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:49:30.714433    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:50:00.715238 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:50:00.715194    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:50:30.715214 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:50:30.715170    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:50:32.257285 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:32.257250    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"]
Nov 03 02:50:32.257905 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:32.257504    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n" podUID="aef0dd90-1756-409d-91f1-43d598c9bf19" containerName="istio-proxy" containerID="cri-o://4c19e2b60a272f10d86ed505257cde1d0d4345532e15aa97517c0b99bd5a98d6" gracePeriod=30
Nov 03 02:50:33.103396 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:33.103357    5005 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 02:50:33.103595 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:33.103421    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n" podUID="aef0dd90-1756-409d-91f1-43d598c9bf19" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 02:50:37.281245 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.281214    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"]
Nov 03 02:50:37.281622 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.281410    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph" podUID="39eeb830-577b-4fcf-9a4b-9d90c15c8be4" containerName="maas-api" containerID="cri-o://0ca278cd34a070be959e9cbe3093f8a0c7509e29f895a22b9fa97c08799fb3bc" gracePeriod=30
Nov 03 02:50:37.589375 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.589351    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:50:37.607502 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.607481    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"
Nov 03 02:50:37.618879 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.618856    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-maas-api-tls\") pod \"39eeb830-577b-4fcf-9a4b-9d90c15c8be4\" (UID: \"39eeb830-577b-4fcf-9a4b-9d90c15c8be4\") "
Nov 03 02:50:37.618981 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.618968    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-data\") pod \"aef0dd90-1756-409d-91f1-43d598c9bf19\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") "
Nov 03 02:50:37.619026 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.619003    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-token\" (UniqueName: \"kubernetes.io/projected/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-token\") pod \"aef0dd90-1756-409d-91f1-43d598c9bf19\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") "
Nov 03 02:50:37.619026 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.619021    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-credential-socket\") pod \"aef0dd90-1756-409d-91f1-43d598c9bf19\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") "
Nov 03 02:50:37.619095 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.619052    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-kube-api-access-fjbf4\" (UniqueName: \"kubernetes.io/projected/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-kube-api-access-fjbf4\") pod \"39eeb830-577b-4fcf-9a4b-9d90c15c8be4\" (UID: \"39eeb830-577b-4fcf-9a4b-9d90c15c8be4\") "
Nov 03 02:50:37.619134 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.619095    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-envoy\") pod \"aef0dd90-1756-409d-91f1-43d598c9bf19\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") "
Nov 03 02:50:37.619134 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.619127    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aef0dd90-1756-409d-91f1-43d598c9bf19-kube-api-access-khwvx\" (UniqueName: \"kubernetes.io/projected/aef0dd90-1756-409d-91f1-43d598c9bf19-kube-api-access-khwvx\") pod \"aef0dd90-1756-409d-91f1-43d598c9bf19\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") "
Nov 03 02:50:37.619236 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.619196    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-podinfo\") pod \"aef0dd90-1756-409d-91f1-43d598c9bf19\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") "
Nov 03 02:50:37.619296 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.619258    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aef0dd90-1756-409d-91f1-43d598c9bf19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aef0dd90-1756-409d-91f1-43d598c9bf19-istiod-ca-cert\") pod \"aef0dd90-1756-409d-91f1-43d598c9bf19\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") "
Nov 03 02:50:37.619358 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.619300    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-workload-socket\") pod \"aef0dd90-1756-409d-91f1-43d598c9bf19\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") "
Nov 03 02:50:37.619358 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.619310    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-data" pod "aef0dd90-1756-409d-91f1-43d598c9bf19" (UID: "aef0dd90-1756-409d-91f1-43d598c9bf19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:50:37.619358 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.619325    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-credential-socket" pod "aef0dd90-1756-409d-91f1-43d598c9bf19" (UID: "aef0dd90-1756-409d-91f1-43d598c9bf19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:50:37.619358 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.619351    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-openshift-service-ca\") pod \"39eeb830-577b-4fcf-9a4b-9d90c15c8be4\" (UID: \"39eeb830-577b-4fcf-9a4b-9d90c15c8be4\") "
Nov 03 02:50:37.619582 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.619414    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-workload-certs\") pod \"aef0dd90-1756-409d-91f1-43d598c9bf19\" (UID: \"aef0dd90-1756-409d-91f1-43d598c9bf19\") "
Nov 03 02:50:37.619582 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.619455    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-metrics-certs\") pod \"39eeb830-577b-4fcf-9a4b-9d90c15c8be4\" (UID: \"39eeb830-577b-4fcf-9a4b-9d90c15c8be4\") "
Nov 03 02:50:37.619709 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.619665    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aef0dd90-1756-409d-91f1-43d598c9bf19-istiod-ca-cert" pod "aef0dd90-1756-409d-91f1-43d598c9bf19" (UID: "aef0dd90-1756-409d-91f1-43d598c9bf19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:50:37.620976 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.620036    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:37.620976 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.620063    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aef0dd90-1756-409d-91f1-43d598c9bf19-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:37.620976 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.620077    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:37.620976 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.620464    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-openshift-service-ca" pod "39eeb830-577b-4fcf-9a4b-9d90c15c8be4" (UID: "39eeb830-577b-4fcf-9a4b-9d90c15c8be4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:50:37.620976 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.620728    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-workload-socket" pod "aef0dd90-1756-409d-91f1-43d598c9bf19" (UID: "aef0dd90-1756-409d-91f1-43d598c9bf19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:50:37.620976 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.620910    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-workload-certs" pod "aef0dd90-1756-409d-91f1-43d598c9bf19" (UID: "aef0dd90-1756-409d-91f1-43d598c9bf19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:50:37.621544 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.621516    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-maas-api-tls" pod "39eeb830-577b-4fcf-9a4b-9d90c15c8be4" (UID: "39eeb830-577b-4fcf-9a4b-9d90c15c8be4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:50:37.621687 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.621648    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-token" pod "aef0dd90-1756-409d-91f1-43d598c9bf19" (UID: "aef0dd90-1756-409d-91f1-43d598c9bf19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:50:37.621886 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.621862    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-kube-api-access-fjbf4" pod "39eeb830-577b-4fcf-9a4b-9d90c15c8be4" (UID: "39eeb830-577b-4fcf-9a4b-9d90c15c8be4"). InnerVolumeSpecName "kube-api-access-fjbf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:50:37.621982 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.621921    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-podinfo" pod "aef0dd90-1756-409d-91f1-43d598c9bf19" (UID: "aef0dd90-1756-409d-91f1-43d598c9bf19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:50:37.621982 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.621973    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-envoy" pod "aef0dd90-1756-409d-91f1-43d598c9bf19" (UID: "aef0dd90-1756-409d-91f1-43d598c9bf19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:50:37.622066 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.622008    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef0dd90-1756-409d-91f1-43d598c9bf19-kube-api-access-khwvx" pod "aef0dd90-1756-409d-91f1-43d598c9bf19" (UID: "aef0dd90-1756-409d-91f1-43d598c9bf19"). InnerVolumeSpecName "kube-api-access-khwvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:50:37.622066 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.622057    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-metrics-certs" pod "39eeb830-577b-4fcf-9a4b-9d90c15c8be4" (UID: "39eeb830-577b-4fcf-9a4b-9d90c15c8be4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:50:37.721581 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.721544    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:37.721581 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.721577    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjbf4\" (UniqueName: \"kubernetes.io/projected/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-kube-api-access-fjbf4\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:37.721581 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.721589    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:37.721865 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.721598    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khwvx\" (UniqueName: \"kubernetes.io/projected/aef0dd90-1756-409d-91f1-43d598c9bf19-kube-api-access-khwvx\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:37.721865 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.721608    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aef0dd90-1756-409d-91f1-43d598c9bf19-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:37.721865 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.721617    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:37.721865 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.721625    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:37.721865 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.721633    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef0dd90-1756-409d-91f1-43d598c9bf19-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:37.721865 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.721642    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:37.721865 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:37.721651    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39eeb830-577b-4fcf-9a4b-9d90c15c8be4-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:38.370383 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:38.370344    5005 generic.go:356] "Generic (PLEG): container finished" podID="aef0dd90-1756-409d-91f1-43d598c9bf19" containerID="4c19e2b60a272f10d86ed505257cde1d0d4345532e15aa97517c0b99bd5a98d6" exitCode=0
Nov 03 02:50:38.370894 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:38.370389    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"
Nov 03 02:50:38.370894 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:38.370428    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n" event={"ID":"aef0dd90-1756-409d-91f1-43d598c9bf19","Type":"ContainerDied","Data":"4c19e2b60a272f10d86ed505257cde1d0d4345532e15aa97517c0b99bd5a98d6"}
Nov 03 02:50:38.370894 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:38.370481    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n" event={"ID":"aef0dd90-1756-409d-91f1-43d598c9bf19","Type":"ContainerDied","Data":"db3e0e3f2968949d1a795bb86328f5bd868be2cefc73290928e761fa6525a12e"}
Nov 03 02:50:38.370894 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:38.370510    5005 scope.go:122] "RemoveContainer" containerID="4c19e2b60a272f10d86ed505257cde1d0d4345532e15aa97517c0b99bd5a98d6"
Nov 03 02:50:38.371932 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:38.371827    5005 generic.go:356] "Generic (PLEG): container finished" podID="39eeb830-577b-4fcf-9a4b-9d90c15c8be4" containerID="0ca278cd34a070be959e9cbe3093f8a0c7509e29f895a22b9fa97c08799fb3bc" exitCode=0
Nov 03 02:50:38.371932 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:38.371875    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"
Nov 03 02:50:38.371932 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:38.371913    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph" event={"ID":"39eeb830-577b-4fcf-9a4b-9d90c15c8be4","Type":"ContainerDied","Data":"0ca278cd34a070be959e9cbe3093f8a0c7509e29f895a22b9fa97c08799fb3bc"}
Nov 03 02:50:38.372069 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:38.371966    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph" event={"ID":"39eeb830-577b-4fcf-9a4b-9d90c15c8be4","Type":"ContainerDied","Data":"706f404ae2908deab17821e32567de73e527ca3b175db3975e0e91d421721d3b"}
Nov 03 02:50:38.382560 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:38.382539    5005 scope.go:122] "RemoveContainer" containerID="4c19e2b60a272f10d86ed505257cde1d0d4345532e15aa97517c0b99bd5a98d6"
Nov 03 02:50:38.382999 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:50:38.382979    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c19e2b60a272f10d86ed505257cde1d0d4345532e15aa97517c0b99bd5a98d6\": container with ID starting with 4c19e2b60a272f10d86ed505257cde1d0d4345532e15aa97517c0b99bd5a98d6 not found: ID does not exist" containerID="4c19e2b60a272f10d86ed505257cde1d0d4345532e15aa97517c0b99bd5a98d6"
Nov 03 02:50:38.383086 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:38.383010    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c19e2b60a272f10d86ed505257cde1d0d4345532e15aa97517c0b99bd5a98d6"} err="failed to get container status \"4c19e2b60a272f10d86ed505257cde1d0d4345532e15aa97517c0b99bd5a98d6\": rpc error: code = NotFound desc = could not find container \"4c19e2b60a272f10d86ed505257cde1d0d4345532e15aa97517c0b99bd5a98d6\": container with ID starting with 4c19e2b60a272f10d86ed505257cde1d0d4345532e15aa97517c0b99bd5a98d6 not found: ID does not exist"
Nov 03 02:50:38.383086 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:38.383034    5005 scope.go:122] "RemoveContainer" containerID="0ca278cd34a070be959e9cbe3093f8a0c7509e29f895a22b9fa97c08799fb3bc"
Nov 03 02:50:38.392082 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:38.392063    5005 scope.go:122] "RemoveContainer" containerID="0ca278cd34a070be959e9cbe3093f8a0c7509e29f895a22b9fa97c08799fb3bc"
Nov 03 02:50:38.392335 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:50:38.392318    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca278cd34a070be959e9cbe3093f8a0c7509e29f895a22b9fa97c08799fb3bc\": container with ID starting with 0ca278cd34a070be959e9cbe3093f8a0c7509e29f895a22b9fa97c08799fb3bc not found: ID does not exist" containerID="0ca278cd34a070be959e9cbe3093f8a0c7509e29f895a22b9fa97c08799fb3bc"
Nov 03 02:50:38.392398 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:38.392346    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca278cd34a070be959e9cbe3093f8a0c7509e29f895a22b9fa97c08799fb3bc"} err="failed to get container status \"0ca278cd34a070be959e9cbe3093f8a0c7509e29f895a22b9fa97c08799fb3bc\": rpc error: code = NotFound desc = could not find container \"0ca278cd34a070be959e9cbe3093f8a0c7509e29f895a22b9fa97c08799fb3bc\": container with ID starting with 0ca278cd34a070be959e9cbe3093f8a0c7509e29f895a22b9fa97c08799fb3bc not found: ID does not exist"
Nov 03 02:50:38.526103 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:38.526058    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"]
Nov 03 02:50:38.572390 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:38.572309    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-vbt7n"]
Nov 03 02:50:38.673552 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:38.673517    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"]
Nov 03 02:50:38.676649 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:38.676625    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ca6ef4-7bf95bb644-jt5ph"]
Nov 03 02:50:39.110340 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:39.110308    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39eeb830-577b-4fcf-9a4b-9d90c15c8be4" path="/var/lib/kubelet/pods/39eeb830-577b-4fcf-9a4b-9d90c15c8be4/volumes"
Nov 03 02:50:39.110905 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:39.110890    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aef0dd90-1756-409d-91f1-43d598c9bf19" path="/var/lib/kubelet/pods/aef0dd90-1756-409d-91f1-43d598c9bf19/volumes"
Nov 03 02:50:48.080785 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.080750    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"]
Nov 03 02:50:48.081197 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.081045    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b" podUID="df56d3fe-d8fa-40a1-80fc-193ea6ed4a29" containerName="maas-api" containerID="cri-o://ad64e9d5ccdac0f8accbcb3c3af8ccbf6a9a6bc0a21c2a6f750048a4bc78ecfd" gracePeriod=30
Nov 03 02:50:48.390119 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.390094    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"
Nov 03 02:50:48.435437 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.435401    5005 generic.go:356] "Generic (PLEG): container finished" podID="df56d3fe-d8fa-40a1-80fc-193ea6ed4a29" containerID="ad64e9d5ccdac0f8accbcb3c3af8ccbf6a9a6bc0a21c2a6f750048a4bc78ecfd" exitCode=0
Nov 03 02:50:48.435600 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.435448    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"
Nov 03 02:50:48.435600 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.435488    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b" event={"ID":"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29","Type":"ContainerDied","Data":"ad64e9d5ccdac0f8accbcb3c3af8ccbf6a9a6bc0a21c2a6f750048a4bc78ecfd"}
Nov 03 02:50:48.435600 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.435557    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b" event={"ID":"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29","Type":"ContainerDied","Data":"15963b5df98947b2b0cbd4cd0862c298229cde49d94cd236829bf2ae90e56c7b"}
Nov 03 02:50:48.435600 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.435588    5005 scope.go:122] "RemoveContainer" containerID="ad64e9d5ccdac0f8accbcb3c3af8ccbf6a9a6bc0a21c2a6f750048a4bc78ecfd"
Nov 03 02:50:48.445629 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.445607    5005 scope.go:122] "RemoveContainer" containerID="ad64e9d5ccdac0f8accbcb3c3af8ccbf6a9a6bc0a21c2a6f750048a4bc78ecfd"
Nov 03 02:50:48.445900 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:50:48.445881    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad64e9d5ccdac0f8accbcb3c3af8ccbf6a9a6bc0a21c2a6f750048a4bc78ecfd\": container with ID starting with ad64e9d5ccdac0f8accbcb3c3af8ccbf6a9a6bc0a21c2a6f750048a4bc78ecfd not found: ID does not exist" containerID="ad64e9d5ccdac0f8accbcb3c3af8ccbf6a9a6bc0a21c2a6f750048a4bc78ecfd"
Nov 03 02:50:48.445963 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.445909    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad64e9d5ccdac0f8accbcb3c3af8ccbf6a9a6bc0a21c2a6f750048a4bc78ecfd"} err="failed to get container status \"ad64e9d5ccdac0f8accbcb3c3af8ccbf6a9a6bc0a21c2a6f750048a4bc78ecfd\": rpc error: code = NotFound desc = could not find container \"ad64e9d5ccdac0f8accbcb3c3af8ccbf6a9a6bc0a21c2a6f750048a4bc78ecfd\": container with ID starting with ad64e9d5ccdac0f8accbcb3c3af8ccbf6a9a6bc0a21c2a6f750048a4bc78ecfd not found: ID does not exist"
Nov 03 02:50:48.454324 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.454306    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-maas-api-tls\") pod \"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29\" (UID: \"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29\") "
Nov 03 02:50:48.454393 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.454369    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-kube-api-access-6pwhz\" (UniqueName: \"kubernetes.io/projected/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-kube-api-access-6pwhz\") pod \"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29\" (UID: \"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29\") "
Nov 03 02:50:48.454444 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.454430    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-metrics-certs\") pod \"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29\" (UID: \"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29\") "
Nov 03 02:50:48.454515 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.454501    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-openshift-service-ca\") pod \"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29\" (UID: \"df56d3fe-d8fa-40a1-80fc-193ea6ed4a29\") "
Nov 03 02:50:48.454864 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.454837    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-openshift-service-ca" pod "df56d3fe-d8fa-40a1-80fc-193ea6ed4a29" (UID: "df56d3fe-d8fa-40a1-80fc-193ea6ed4a29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:50:48.455051 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.455035    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:48.456433 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.456411    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-metrics-certs" pod "df56d3fe-d8fa-40a1-80fc-193ea6ed4a29" (UID: "df56d3fe-d8fa-40a1-80fc-193ea6ed4a29"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:50:48.456887 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.456860    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-kube-api-access-6pwhz" pod "df56d3fe-d8fa-40a1-80fc-193ea6ed4a29" (UID: "df56d3fe-d8fa-40a1-80fc-193ea6ed4a29"). InnerVolumeSpecName "kube-api-access-6pwhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:50:48.456957 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.456878    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-maas-api-tls" pod "df56d3fe-d8fa-40a1-80fc-193ea6ed4a29" (UID: "df56d3fe-d8fa-40a1-80fc-193ea6ed4a29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:50:48.556322 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.556278    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pwhz\" (UniqueName: \"kubernetes.io/projected/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-kube-api-access-6pwhz\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:48.556322 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.556319    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:48.556527 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.556336    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:48.850317 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.850273    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"]
Nov 03 02:50:48.854421 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:48.854395    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-604da0-68f5874cbd-fz74b"]
Nov 03 02:50:49.127151 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:50:49.127108    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df56d3fe-d8fa-40a1-80fc-193ea6ed4a29" path="/var/lib/kubelet/pods/df56d3fe-d8fa-40a1-80fc-193ea6ed4a29/volumes"
Nov 03 02:51:00.713503 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:51:00.713461    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:51:13.635687 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:13.635659    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:51:13.647217 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:13.647195    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:51:30.714722 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:51:30.714658    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:51:45.169420 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.169382    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"]
Nov 03 02:51:45.170244 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.170020    5005 cpu_manager.go:414] "containerMap: removing container" podUID="df56d3fe-d8fa-40a1-80fc-193ea6ed4a29" containerName="maas-api"
Nov 03 02:51:45.170244 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.170034    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df56d3fe-d8fa-40a1-80fc-193ea6ed4a29" containerName="maas-api"
Nov 03 02:51:45.170244 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.170063    5005 cpu_manager.go:414] "containerMap: removing container" podUID="39eeb830-577b-4fcf-9a4b-9d90c15c8be4" containerName="maas-api"
Nov 03 02:51:45.170244 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.170068    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39eeb830-577b-4fcf-9a4b-9d90c15c8be4" containerName="maas-api"
Nov 03 02:51:45.170244 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.170076    5005 cpu_manager.go:414] "containerMap: removing container" podUID="aef0dd90-1756-409d-91f1-43d598c9bf19" containerName="istio-proxy"
Nov 03 02:51:45.170244 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.170081    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aef0dd90-1756-409d-91f1-43d598c9bf19" containerName="istio-proxy"
Nov 03 02:51:45.170244 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.170179    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="39eeb830-577b-4fcf-9a4b-9d90c15c8be4" containerName="maas-api"
Nov 03 02:51:45.170244 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.170194    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="aef0dd90-1756-409d-91f1-43d598c9bf19" containerName="istio-proxy"
Nov 03 02:51:45.170244 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.170206    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="df56d3fe-d8fa-40a1-80fc-193ea6ed4a29" containerName="maas-api"
Nov 03 02:51:45.173334 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.173312    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"
Nov 03 02:51:45.228019 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.227996    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-e039cb-serving-cert\""
Nov 03 02:51:45.278892 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.278854    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"]
Nov 03 02:51:45.332464 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.332433    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2s5c\" (UniqueName: \"kubernetes.io/projected/22dc475f-cfa4-435c-b021-ccdd2f2eec91-kube-api-access-q2s5c\") pod \"maas-api-e2e-extmod-a-e039cb-846897867f-rs926\" (UID: \"22dc475f-cfa4-435c-b021-ccdd2f2eec91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"
Nov 03 02:51:45.332643 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.332484    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22dc475f-cfa4-435c-b021-ccdd2f2eec91-metrics-certs\") pod \"maas-api-e2e-extmod-a-e039cb-846897867f-rs926\" (UID: \"22dc475f-cfa4-435c-b021-ccdd2f2eec91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"
Nov 03 02:51:45.332643 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.332572    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22dc475f-cfa4-435c-b021-ccdd2f2eec91-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e039cb-846897867f-rs926\" (UID: \"22dc475f-cfa4-435c-b021-ccdd2f2eec91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"
Nov 03 02:51:45.332803 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.332784    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22dc475f-cfa4-435c-b021-ccdd2f2eec91-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e039cb-846897867f-rs926\" (UID: \"22dc475f-cfa4-435c-b021-ccdd2f2eec91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"
Nov 03 02:51:45.434599 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.434508    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2s5c\" (UniqueName: \"kubernetes.io/projected/22dc475f-cfa4-435c-b021-ccdd2f2eec91-kube-api-access-q2s5c\") pod \"maas-api-e2e-extmod-a-e039cb-846897867f-rs926\" (UID: \"22dc475f-cfa4-435c-b021-ccdd2f2eec91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"
Nov 03 02:51:45.434599 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.434558    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22dc475f-cfa4-435c-b021-ccdd2f2eec91-metrics-certs\") pod \"maas-api-e2e-extmod-a-e039cb-846897867f-rs926\" (UID: \"22dc475f-cfa4-435c-b021-ccdd2f2eec91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"
Nov 03 02:51:45.434599 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.434579    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22dc475f-cfa4-435c-b021-ccdd2f2eec91-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e039cb-846897867f-rs926\" (UID: \"22dc475f-cfa4-435c-b021-ccdd2f2eec91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"
Nov 03 02:51:45.434945 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.434748    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22dc475f-cfa4-435c-b021-ccdd2f2eec91-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e039cb-846897867f-rs926\" (UID: \"22dc475f-cfa4-435c-b021-ccdd2f2eec91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"
Nov 03 02:51:45.434945 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:51:45.434778    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-serving-cert: secret "maas-api-e2e-extmod-a-e039cb-serving-cert" not found
Nov 03 02:51:45.434945 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:51:45.434866    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22dc475f-cfa4-435c-b021-ccdd2f2eec91-maas-api-tls podName:22dc475f-cfa4-435c-b021-ccdd2f2eec91 nodeName:}" failed. No retries permitted until 2026-11-03 02:51:45.934846105 +0000 UTC m=+3632.949087467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/22dc475f-cfa4-435c-b021-ccdd2f2eec91-maas-api-tls") pod "maas-api-e2e-extmod-a-e039cb-846897867f-rs926" (UID: "22dc475f-cfa4-435c-b021-ccdd2f2eec91") : secret "maas-api-e2e-extmod-a-e039cb-serving-cert" not found
Nov 03 02:51:45.435464 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.435439    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22dc475f-cfa4-435c-b021-ccdd2f2eec91-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e039cb-846897867f-rs926\" (UID: \"22dc475f-cfa4-435c-b021-ccdd2f2eec91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"
Nov 03 02:51:45.437074 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.437053    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22dc475f-cfa4-435c-b021-ccdd2f2eec91-metrics-certs\") pod \"maas-api-e2e-extmod-a-e039cb-846897867f-rs926\" (UID: \"22dc475f-cfa4-435c-b021-ccdd2f2eec91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"
Nov 03 02:51:45.496766 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.496739    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2s5c\" (UniqueName: \"kubernetes.io/projected/22dc475f-cfa4-435c-b021-ccdd2f2eec91-kube-api-access-q2s5c\") pod \"maas-api-e2e-extmod-a-e039cb-846897867f-rs926\" (UID: \"22dc475f-cfa4-435c-b021-ccdd2f2eec91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"
Nov 03 02:51:45.941315 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.941280    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22dc475f-cfa4-435c-b021-ccdd2f2eec91-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e039cb-846897867f-rs926\" (UID: \"22dc475f-cfa4-435c-b021-ccdd2f2eec91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"
Nov 03 02:51:45.943678 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:45.943647    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22dc475f-cfa4-435c-b021-ccdd2f2eec91-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e039cb-846897867f-rs926\" (UID: \"22dc475f-cfa4-435c-b021-ccdd2f2eec91\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"
Nov 03 02:51:46.086890 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:46.086857    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"
Nov 03 02:51:46.462535 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:46.462503    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"]
Nov 03 02:51:46.470929 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:51:46.470892    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22dc475f_cfa4_435c_b021_ccdd2f2eec91.slice/crio-c6487e6331f1b1230f1e7b72e076c3d39417996b3f0e6197d079d0b4167d2281 WatchSource:0}: Error finding container c6487e6331f1b1230f1e7b72e076c3d39417996b3f0e6197d079d0b4167d2281: Status 404 returned error can't find the container with id c6487e6331f1b1230f1e7b72e076c3d39417996b3f0e6197d079d0b4167d2281
Nov 03 02:51:46.812200 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:46.812158    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926" event={"ID":"22dc475f-cfa4-435c-b021-ccdd2f2eec91","Type":"ContainerStarted","Data":"c6487e6331f1b1230f1e7b72e076c3d39417996b3f0e6197d079d0b4167d2281"}
Nov 03 02:51:47.820030 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:47.819992    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926" event={"ID":"22dc475f-cfa4-435c-b021-ccdd2f2eec91","Type":"ContainerStarted","Data":"1b254586f73e1f779cc73fea05b98b5f9eea96775cbd9222f548ae39e3f8526b"}
Nov 03 02:51:47.820430 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:47.820058    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"
Nov 03 02:51:47.979543 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:47.979441    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926" podStartSLOduration=1.7584229439999999 podStartE2EDuration="2.979426297s" totalImagesPullingTime="1.221003353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:51:47.975972841 +0000 UTC m=+3634.990214206" watchObservedRunningTime="2026-11-03 02:51:47.979426297 +0000 UTC m=+3634.993667664"
Nov 03 02:51:53.831507 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:51:53.831431    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"
Nov 03 02:52:00.714545 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:52:00.714507    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:52:30.717524 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:52:30.717486    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:52:51.968263 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:51.968231    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"]
Nov 03 02:52:51.972548 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:51.972528    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.058817 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.058780    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.059017 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.058833    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.059017 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.058923    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.059143 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.059015    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.059143 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.059073    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.059224 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.059147    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gc4g\" (UniqueName: \"kubernetes.io/projected/55f0246d-7f9d-4581-bd2b-c98e20e97676-kube-api-access-2gc4g\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.059224 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.059182    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.059335 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.059269    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f0246d-7f9d-4581-bd2b-c98e20e97676-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.059391 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.059363    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.160245 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.160211    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.160428 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.160272    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gc4g\" (UniqueName: \"kubernetes.io/projected/55f0246d-7f9d-4581-bd2b-c98e20e97676-kube-api-access-2gc4g\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.160428 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.160324    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.160428 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.160361    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f0246d-7f9d-4581-bd2b-c98e20e97676-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.160428 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.160419    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.160654 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.160516    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.160654 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.160564    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.160654 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.160598    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.160654 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.160648    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.160896 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.160672    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.160896 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.160844    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.161009 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.160926    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.161009 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.160991    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.161245 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.161210    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f0246d-7f9d-4581-bd2b-c98e20e97676-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.163058 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.163034    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.163458 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.163440    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.167525 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.167478    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"]
Nov 03 02:52:52.215038 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.215011    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.215169 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.215079    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gc4g\" (UniqueName: \"kubernetes.io/projected/55f0246d-7f9d-4581-bd2b-c98e20e97676-kube-api-access-2gc4g\") pod \"maas-default-gateway-openshift-default-5bc884dd87-99swt\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.285841 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.285815    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:52.715772 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.715671    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"]
Nov 03 02:52:52.721107 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:52:52.721071    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f0246d_7f9d_4581_bd2b_c98e20e97676.slice/crio-c9c158965c9c96b95021f90dac913332a10ebfe594076ec2fd5451c55bdd2009 WatchSource:0}: Error finding container c9c158965c9c96b95021f90dac913332a10ebfe594076ec2fd5451c55bdd2009: Status 404 returned error can't find the container with id c9c158965c9c96b95021f90dac913332a10ebfe594076ec2fd5451c55bdd2009
Nov 03 02:52:52.723658 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.723624    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:52:52.723751 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.723717    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:52:52.723806 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:52.723754    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:52:53.247131 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:53.247091    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt" event={"ID":"55f0246d-7f9d-4581-bd2b-c98e20e97676","Type":"ContainerStarted","Data":"7a5d27b54f44e58debd75bb5b3db172a4c457812c2613d8e54bd0e773c0cacf5"}
Nov 03 02:52:53.247652 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:53.247149    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt" event={"ID":"55f0246d-7f9d-4581-bd2b-c98e20e97676","Type":"ContainerStarted","Data":"c9c158965c9c96b95021f90dac913332a10ebfe594076ec2fd5451c55bdd2009"}
Nov 03 02:52:53.290458 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:53.290415    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:52:53.292750 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:53.292297    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-99swt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 03 02:52:53.292750 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:53.292354    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt" podUID="55f0246d-7f9d-4581-bd2b-c98e20e97676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 03 02:52:53.440557 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:53.440497    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt" podStartSLOduration=2.440480474 podStartE2EDuration="2.440480474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:52:53.40931507 +0000 UTC m=+3700.423556467" watchObservedRunningTime="2026-11-03 02:52:53.440480474 +0000 UTC m=+3700.454721839"
Nov 03 02:52:54.286383 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:54.286332    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-99swt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 03 02:52:54.286844 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:54.286431    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt" podUID="55f0246d-7f9d-4581-bd2b-c98e20e97676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 03 02:52:55.286676 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:55.286641    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-99swt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 03 02:52:55.287109 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:55.286727    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt" podUID="55f0246d-7f9d-4581-bd2b-c98e20e97676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 03 02:52:56.286554 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:56.286520    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-99swt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 03 02:52:56.286748 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:56.286573    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt" podUID="55f0246d-7f9d-4581-bd2b-c98e20e97676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 03 02:52:57.286417 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:57.286381    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-99swt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 03 02:52:57.286594 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:57.286439    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt" podUID="55f0246d-7f9d-4581-bd2b-c98e20e97676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 03 02:52:58.286566 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:58.286528    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-99swt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 03 02:52:58.287013 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:58.286600    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt" podUID="55f0246d-7f9d-4581-bd2b-c98e20e97676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 03 02:52:59.286851 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:59.286816    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-99swt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 03 02:52:59.287258 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:52:59.286878    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt" podUID="55f0246d-7f9d-4581-bd2b-c98e20e97676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 03 02:53:00.286286 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:00.286248    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-99swt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 03 02:53:00.286454 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:00.286312    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt" podUID="55f0246d-7f9d-4581-bd2b-c98e20e97676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 03 02:53:00.717882 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:53:00.717801    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:53:01.393359 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:01.393328    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:53:01.394818 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:01.394791    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:53:01.395618 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:01.395596    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 02:53:30.718024 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:53:30.717932    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:53:48.653228 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:48.653197    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"]
Nov 03 02:53:48.657424 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:48.657402    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"
Nov 03 02:53:48.706043 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:48.706016    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ecdbc7-serving-cert\""
Nov 03 02:53:48.770717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:48.768920    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8pxc\" (UniqueName: \"kubernetes.io/projected/f2b37f9f-b995-403b-8ba8-e27273057f9f-kube-api-access-v8pxc\") pod \"maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9\" (UID: \"f2b37f9f-b995-403b-8ba8-e27273057f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"
Nov 03 02:53:48.770717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:48.769020    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b37f9f-b995-403b-8ba8-e27273057f9f-maas-api-tls\") pod \"maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9\" (UID: \"f2b37f9f-b995-403b-8ba8-e27273057f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"
Nov 03 02:53:48.770717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:48.769045    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b37f9f-b995-403b-8ba8-e27273057f9f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9\" (UID: \"f2b37f9f-b995-403b-8ba8-e27273057f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"
Nov 03 02:53:48.770717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:48.769220    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b37f9f-b995-403b-8ba8-e27273057f9f-metrics-certs\") pod \"maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9\" (UID: \"f2b37f9f-b995-403b-8ba8-e27273057f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"
Nov 03 02:53:48.795714 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:48.794127    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"]
Nov 03 02:53:48.870188 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:48.870155    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b37f9f-b995-403b-8ba8-e27273057f9f-maas-api-tls\") pod \"maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9\" (UID: \"f2b37f9f-b995-403b-8ba8-e27273057f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"
Nov 03 02:53:48.870188 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:48.870191    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b37f9f-b995-403b-8ba8-e27273057f9f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9\" (UID: \"f2b37f9f-b995-403b-8ba8-e27273057f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"
Nov 03 02:53:48.870436 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:53:48.870287    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-serving-cert: secret "maas-api-e2e-inf-a-ecdbc7-serving-cert" not found
Nov 03 02:53:48.870436 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:48.870296    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b37f9f-b995-403b-8ba8-e27273057f9f-metrics-certs\") pod \"maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9\" (UID: \"f2b37f9f-b995-403b-8ba8-e27273057f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"
Nov 03 02:53:48.870436 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:53:48.870347    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2b37f9f-b995-403b-8ba8-e27273057f9f-maas-api-tls podName:f2b37f9f-b995-403b-8ba8-e27273057f9f nodeName:}" failed. No retries permitted until 2026-11-03 02:53:49.370331555 +0000 UTC m=+3756.384572899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2b37f9f-b995-403b-8ba8-e27273057f9f-maas-api-tls") pod "maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9" (UID: "f2b37f9f-b995-403b-8ba8-e27273057f9f") : secret "maas-api-e2e-inf-a-ecdbc7-serving-cert" not found
Nov 03 02:53:48.870436 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:48.870399    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8pxc\" (UniqueName: \"kubernetes.io/projected/f2b37f9f-b995-403b-8ba8-e27273057f9f-kube-api-access-v8pxc\") pod \"maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9\" (UID: \"f2b37f9f-b995-403b-8ba8-e27273057f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"
Nov 03 02:53:48.871019 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:48.871000    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b37f9f-b995-403b-8ba8-e27273057f9f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9\" (UID: \"f2b37f9f-b995-403b-8ba8-e27273057f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"
Nov 03 02:53:48.872907 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:48.872880    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b37f9f-b995-403b-8ba8-e27273057f9f-metrics-certs\") pod \"maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9\" (UID: \"f2b37f9f-b995-403b-8ba8-e27273057f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"
Nov 03 02:53:48.961817 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:48.942924    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8pxc\" (UniqueName: \"kubernetes.io/projected/f2b37f9f-b995-403b-8ba8-e27273057f9f-kube-api-access-v8pxc\") pod \"maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9\" (UID: \"f2b37f9f-b995-403b-8ba8-e27273057f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"
Nov 03 02:53:49.377165 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:49.377130    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b37f9f-b995-403b-8ba8-e27273057f9f-maas-api-tls\") pod \"maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9\" (UID: \"f2b37f9f-b995-403b-8ba8-e27273057f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"
Nov 03 02:53:49.379527 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:49.379497    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b37f9f-b995-403b-8ba8-e27273057f9f-maas-api-tls\") pod \"maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9\" (UID: \"f2b37f9f-b995-403b-8ba8-e27273057f9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"
Nov 03 02:53:49.571392 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:49.571352    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"
Nov 03 02:53:49.942725 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:49.942622    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"]
Nov 03 02:53:49.948969 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:53:49.948926    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b37f9f_b995_403b_8ba8_e27273057f9f.slice/crio-95d12329a596dedfc94b27c2b52f7e62c2be541c83e5a086ad53c68c8d3dc9bc WatchSource:0}: Error finding container 95d12329a596dedfc94b27c2b52f7e62c2be541c83e5a086ad53c68c8d3dc9bc: Status 404 returned error can't find the container with id 95d12329a596dedfc94b27c2b52f7e62c2be541c83e5a086ad53c68c8d3dc9bc
Nov 03 02:53:49.950452 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:49.950434    5005 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 02:53:50.659284 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:50.659252    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9" event={"ID":"f2b37f9f-b995-403b-8ba8-e27273057f9f","Type":"ContainerStarted","Data":"95d12329a596dedfc94b27c2b52f7e62c2be541c83e5a086ad53c68c8d3dc9bc"}
Nov 03 02:53:51.690073 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:51.690031    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9" event={"ID":"f2b37f9f-b995-403b-8ba8-e27273057f9f","Type":"ContainerStarted","Data":"ec06bd451c3194bbade2e1e3088afc24fb8839fb13747b2e3c4d411f2d576bec"}
Nov 03 02:53:51.690655 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:51.690624    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"
Nov 03 02:53:51.801174 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:51.801123    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9" podStartSLOduration=2.80484284 podStartE2EDuration="3.801109954s" totalImagesPullingTime="996.267114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:53:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:53:51.799253008 +0000 UTC m=+3758.813494375" watchObservedRunningTime="2026-11-03 02:53:51.801109954 +0000 UTC m=+3758.815351319"
Nov 03 02:53:52.390752 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.390720    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"]
Nov 03 02:53:52.395194 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.395176    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.410038 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.410013    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.410157 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.410062    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700d764b-0150-4711-aa7c-718170d233d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.410157 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.410139    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kvm5\" (UniqueName: \"kubernetes.io/projected/700d764b-0150-4711-aa7c-718170d233d5-kube-api-access-4kvm5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.410292 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.410272    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.410351 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.410308    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.410407 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.410363    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700d764b-0150-4711-aa7c-718170d233d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.410453 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.410443    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.410492 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.410471    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700d764b-0150-4711-aa7c-718170d233d5-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.410543 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.410496    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.511654 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.511619    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700d764b-0150-4711-aa7c-718170d233d5-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.511846 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.511665    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.511846 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.511727    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.511846 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.511769    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700d764b-0150-4711-aa7c-718170d233d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.511846 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.511799    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kvm5\" (UniqueName: \"kubernetes.io/projected/700d764b-0150-4711-aa7c-718170d233d5-kube-api-access-4kvm5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.512080 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.511900    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.512080 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.511919    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.512080 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.511971    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700d764b-0150-4711-aa7c-718170d233d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.512080 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.512035    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.512296 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.512204    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.512296 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.512228    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.512393 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.512380    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.512483 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.512460    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.512857 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.512836    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700d764b-0150-4711-aa7c-718170d233d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.514308 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.514281    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.514559 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.514538    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700d764b-0150-4711-aa7c-718170d233d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.565658 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.565621    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700d764b-0150-4711-aa7c-718170d233d5-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.565658 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.565650    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kvm5\" (UniqueName: \"kubernetes.io/projected/700d764b-0150-4711-aa7c-718170d233d5-kube-api-access-4kvm5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7fn2w\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:52.584547 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.584500    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"]
Nov 03 02:53:52.709220 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:52.709138    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:53.138601 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:53.138568    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"]
Nov 03 02:53:53.183872 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:53:53.183845    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700d764b_0150_4711_aa7c_718170d233d5.slice/crio-201ca749f646501bdf57f2abf737c6310de8609540b1688896098bd9fad9ae23 WatchSource:0}: Error finding container 201ca749f646501bdf57f2abf737c6310de8609540b1688896098bd9fad9ae23: Status 404 returned error can't find the container with id 201ca749f646501bdf57f2abf737c6310de8609540b1688896098bd9fad9ae23
Nov 03 02:53:53.186520 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:53.186484    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:53:53.186597 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:53.186556    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:53:53.186597 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:53.186588    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:53:53.704563 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:53.704520    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w" event={"ID":"700d764b-0150-4711-aa7c-718170d233d5","Type":"ContainerStarted","Data":"a476974720c576cb6b85b386a528ca2c0c5a97cc903169c280b08bf4a18ee733"}
Nov 03 02:53:53.704796 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:53.704581    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w" event={"ID":"700d764b-0150-4711-aa7c-718170d233d5","Type":"ContainerStarted","Data":"201ca749f646501bdf57f2abf737c6310de8609540b1688896098bd9fad9ae23"}
Nov 03 02:53:53.709624 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:53.709599    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:53:53.817869 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:53.817820    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w" podStartSLOduration=1.817805764 podStartE2EDuration="1.817805764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:53:53.813485587 +0000 UTC m=+3760.827726953" watchObservedRunningTime="2026-11-03 02:53:53.817805764 +0000 UTC m=+3760.832047190"
Nov 03 02:53:54.726523 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:54.726484    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7fn2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 03 02:53:54.727089 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:54.726540    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w" podUID="700d764b-0150-4711-aa7c-718170d233d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 03 02:53:55.722721 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:55.722062    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7fn2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 03 02:53:55.722721 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:55.722138    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w" podUID="700d764b-0150-4711-aa7c-718170d233d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 03 02:53:56.719718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:56.719109    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7fn2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 03 02:53:56.719718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:56.719174    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w" podUID="700d764b-0150-4711-aa7c-718170d233d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 03 02:53:57.714505 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:57.714466    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7fn2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 03 02:53:57.714889 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:57.714857    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w" podUID="700d764b-0150-4711-aa7c-718170d233d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 03 02:53:57.724773 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:57.724741    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"
Nov 03 02:53:58.718718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:58.718102    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7fn2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 03 02:53:58.718718 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:58.718181    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w" podUID="700d764b-0150-4711-aa7c-718170d233d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 03 02:53:59.710026 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:59.709993    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7fn2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 03 02:53:59.710426 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:53:59.710072    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w" podUID="700d764b-0150-4711-aa7c-718170d233d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 03 02:54:00.709805 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:54:00.709769    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7fn2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 03 02:54:00.709992 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:54:00.709835    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w" podUID="700d764b-0150-4711-aa7c-718170d233d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 03 02:54:00.715646 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:54:00.715609    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:54:02.054292 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:54:02.054151    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:54:02.055484 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:54:02.055457    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:54:02.056867 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:54:02.056845    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 02:54:30.714637 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:54:30.714595    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:55:00.713560 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:55:00.713476    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:55:12.509041 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.509008    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"]
Nov 03 02:55:12.513288 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.513269    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.589184 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.589147    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.589348 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.589192    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.589415 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.589342    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.589484 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.589445    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a37751f-fd21-4160-a44f-e97a796f9198-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.589551 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.589535    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.589614 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.589600    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a37751f-fd21-4160-a44f-e97a796f9198-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.589665 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.589651    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.589734 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.589718    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwnz6\" (UniqueName: \"kubernetes.io/projected/6a37751f-fd21-4160-a44f-e97a796f9198-kube-api-access-gwnz6\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.589791 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.589751    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a37751f-fd21-4160-a44f-e97a796f9198-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.605517 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.605486    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"]
Nov 03 02:55:12.648495 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.648459    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-dc14f6-openshift-default-dockercfg-m277s\""
Nov 03 02:55:12.691003 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.690968    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.691173 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.691010    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.691173 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.691058    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.691173 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.691112    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a37751f-fd21-4160-a44f-e97a796f9198-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.691173 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.691167    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.691401 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.691231    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a37751f-fd21-4160-a44f-e97a796f9198-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.691401 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.691280    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.691519 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.691354    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwnz6\" (UniqueName: \"kubernetes.io/projected/6a37751f-fd21-4160-a44f-e97a796f9198-kube-api-access-gwnz6\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.691519 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.691434    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.691519 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.691460    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a37751f-fd21-4160-a44f-e97a796f9198-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.691519 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.691483    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.691746 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.691581    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.691812 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.691790    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.692044 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.692023    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a37751f-fd21-4160-a44f-e97a796f9198-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.693599 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.693577    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.693772 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.693756    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a37751f-fd21-4160-a44f-e97a796f9198-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.747029 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.746998    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a37751f-fd21-4160-a44f-e97a796f9198-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.747255 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.747229    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwnz6\" (UniqueName: \"kubernetes.io/projected/6a37751f-fd21-4160-a44f-e97a796f9198-kube-api-access-gwnz6\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:12.827471 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:12.827243    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:13.259523 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:13.259486    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"]
Nov 03 02:55:13.266294 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:55:13.266265    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a37751f_fd21_4160_a44f_e97a796f9198.slice/crio-b73f948d6fe1315069962d93b3cc693311c1f54a58bab9917cb4d9baa36e125e WatchSource:0}: Error finding container b73f948d6fe1315069962d93b3cc693311c1f54a58bab9917cb4d9baa36e125e: Status 404 returned error can't find the container with id b73f948d6fe1315069962d93b3cc693311c1f54a58bab9917cb4d9baa36e125e
Nov 03 02:55:13.269205 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:13.269170    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:55:13.269298 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:13.269243    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:55:13.269298 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:13.269273    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:55:13.364206 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:13.363793    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc" event={"ID":"6a37751f-fd21-4160-a44f-e97a796f9198","Type":"ContainerStarted","Data":"8f80e6ed396306f9351ec492c37fd65594d0a20d880be87e108ef96185069a8a"}
Nov 03 02:55:13.364206 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:13.363877    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc" event={"ID":"6a37751f-fd21-4160-a44f-e97a796f9198","Type":"ContainerStarted","Data":"b73f948d6fe1315069962d93b3cc693311c1f54a58bab9917cb4d9baa36e125e"}
Nov 03 02:55:13.472819 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:13.472736    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc" podStartSLOduration=1.472688456 podStartE2EDuration="1.472688456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:55:13.472645699 +0000 UTC m=+3840.486887090" watchObservedRunningTime="2026-11-03 02:55:13.472688456 +0000 UTC m=+3840.486929823"
Nov 03 02:55:13.829093 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:13.829042    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:14.830036 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:14.829996    5005 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 02:55:14.830442 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:14.830053    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc" podUID="6a37751f-fd21-4160-a44f-e97a796f9198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 02:55:15.828442 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:15.828403    5005 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 02:55:15.828707 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:15.828668    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc" podUID="6a37751f-fd21-4160-a44f-e97a796f9198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 02:55:16.828238 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:16.828196    5005 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 02:55:16.828663 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:16.828252    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc" podUID="6a37751f-fd21-4160-a44f-e97a796f9198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 02:55:17.827988 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:17.827945    5005 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 02:55:17.828178 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:17.828011    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc" podUID="6a37751f-fd21-4160-a44f-e97a796f9198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 02:55:18.827952 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:18.827915    5005 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 02:55:18.828461 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:18.828086    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc" podUID="6a37751f-fd21-4160-a44f-e97a796f9198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 02:55:19.833665 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:19.833635    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:19.834148 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:19.834128    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:19.834913 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:19.834890    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 02:55:30.715997 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:55:30.715959    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:55:42.751321 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.751289    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"]
Nov 03 02:55:42.757040 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.757018    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.863460 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.863389    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b80d8879-30b7-477c-8b8e-5c806d813e54-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.863661 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.863523    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.863661 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.863593    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.863661 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.863624    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.863912 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.863683    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.863912 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.863719    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.863912 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.863834    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcfms\" (UniqueName: \"kubernetes.io/projected/b80d8879-30b7-477c-8b8e-5c806d813e54-kube-api-access-gcfms\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.864093 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.864015    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.864159 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.864121    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.947282 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.947244    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"]
Nov 03 02:55:42.965950 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.965911    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b80d8879-30b7-477c-8b8e-5c806d813e54-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.966176 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.966005    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.966176 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.966043    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.966176 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.966074    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.966176 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.966138    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.966176 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.966164    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.966478 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.966215    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcfms\" (UniqueName: \"kubernetes.io/projected/b80d8879-30b7-477c-8b8e-5c806d813e54-kube-api-access-gcfms\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.966478 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.966253    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.966478 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.966357    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.966640 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.966470    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-workload-certs\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.966640 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.966501    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-workload-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.966640 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.966567    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-data\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.966833 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.966740    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-credential-socket\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.966833 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.966814    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b80d8879-30b7-477c-8b8e-5c806d813e54-istiod-ca-cert\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.968673 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.968640    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-envoy\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:42.969277 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:42.969252    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-podinfo\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:43.019609 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:43.019574    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-token\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:43.019799 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:43.019659    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcfms\" (UniqueName: \"kubernetes.io/projected/b80d8879-30b7-477c-8b8e-5c806d813e54-kube-api-access-gcfms\") pod \"e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") " pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:43.072328 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:43.072295    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:43.511052 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:43.511013    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"]
Nov 03 02:55:43.516004 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:55:43.515974    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb80d8879_30b7_477c_8b8e_5c806d813e54.slice/crio-f76af134dac86d2adaf02e8a3a282c6c56b797f462011539a6cc8b672bc71bc4 WatchSource:0}: Error finding container f76af134dac86d2adaf02e8a3a282c6c56b797f462011539a6cc8b672bc71bc4: Status 404 returned error can't find the container with id f76af134dac86d2adaf02e8a3a282c6c56b797f462011539a6cc8b672bc71bc4
Nov 03 02:55:43.518708 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:43.518660    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:55:43.518810 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:43.518757    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:55:43.518810 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:43.518794    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 02:55:43.781953 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:43.781855    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv" event={"ID":"b80d8879-30b7-477c-8b8e-5c806d813e54","Type":"ContainerStarted","Data":"c45cb38dc33257fbd4aa89532221683af3b5ca4bb856c3d257f1610dd5f000dd"}
Nov 03 02:55:43.781953 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:43.781917    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv" event={"ID":"b80d8879-30b7-477c-8b8e-5c806d813e54","Type":"ContainerStarted","Data":"f76af134dac86d2adaf02e8a3a282c6c56b797f462011539a6cc8b672bc71bc4"}
Nov 03 02:55:44.077530 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:44.077439    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:44.077708 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:44.077643    5005 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 02:55:44.077781 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:44.077737    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv" podUID="b80d8879-30b7-477c-8b8e-5c806d813e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 02:55:45.072865 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:45.072826    5005 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 02:55:45.073287 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:45.072882    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv" podUID="b80d8879-30b7-477c-8b8e-5c806d813e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 02:55:46.073621 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:46.073587    5005 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 02:55:46.074062 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:46.073651    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv" podUID="b80d8879-30b7-477c-8b8e-5c806d813e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 02:55:47.072859 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:47.072820    5005 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 02:55:47.073053 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:47.072887    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv" podUID="b80d8879-30b7-477c-8b8e-5c806d813e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 02:55:48.073044 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:48.073008    5005 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 02:55:48.073456 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:48.073070    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv" podUID="b80d8879-30b7-477c-8b8e-5c806d813e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 02:55:49.072802 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:49.072764    5005 patch_prober.go:29] interesting pod/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 02:55:49.073088 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:49.072820    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv" podUID="b80d8879-30b7-477c-8b8e-5c806d813e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 02:55:50.078492 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:50.078457    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:50.079020 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:50.078804    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:50.079506 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:50.079485    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 02:55:50.237081 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:55:50.237029    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv" podStartSLOduration=8.237013299000001 podStartE2EDuration="8.237013299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:55:43.93730284 +0000 UTC m=+3870.951544208" watchObservedRunningTime="2026-11-03 02:55:50.237013299 +0000 UTC m=+3877.251254666"
Nov 03 02:56:00.714746 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:56:00.714705    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:56:13.679189 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:13.679073    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:56:13.693188 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:13.693162    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 02:56:15.293053 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:15.293017    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"]
Nov 03 02:56:15.297932 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:15.297901    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"
Nov 03 02:56:15.347414 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:15.347386    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1c43e9-serving-cert\""
Nov 03 02:56:15.397661 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:15.397623    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"]
Nov 03 02:56:15.399750 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:15.399726    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-metrics-certs\") pod \"maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx\" (UID: \"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"
Nov 03 02:56:15.399925 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:15.399758    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-maas-api-tls\") pod \"maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx\" (UID: \"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"
Nov 03 02:56:15.400005 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:15.399949    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx\" (UID: \"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"
Nov 03 02:56:15.400186 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:15.400159    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmcqs\" (UniqueName: \"kubernetes.io/projected/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-kube-api-access-tmcqs\") pod \"maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx\" (UID: \"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"
Nov 03 02:56:15.501346 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:15.501313    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmcqs\" (UniqueName: \"kubernetes.io/projected/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-kube-api-access-tmcqs\") pod \"maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx\" (UID: \"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"
Nov 03 02:56:15.501523 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:15.501383    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-metrics-certs\") pod \"maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx\" (UID: \"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"
Nov 03 02:56:15.501523 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:15.501407    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-maas-api-tls\") pod \"maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx\" (UID: \"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"
Nov 03 02:56:15.501523 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:15.501492    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx\" (UID: \"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"
Nov 03 02:56:15.501715 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:56:15.501598    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-serving-cert: secret "maas-api-e2e-inf-a-1c43e9-serving-cert" not found
Nov 03 02:56:15.501715 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:56:15.501689    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-maas-api-tls podName:a0b69c55-a8e5-4fb9-8e61-e8533ee739a2 nodeName:}" failed. No retries permitted until 2026-11-03 02:56:16.001666265 +0000 UTC m=+3903.015907628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-maas-api-tls") pod "maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx" (UID: "a0b69c55-a8e5-4fb9-8e61-e8533ee739a2") : secret "maas-api-e2e-inf-a-1c43e9-serving-cert" not found
Nov 03 02:56:15.502254 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:15.502232    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx\" (UID: \"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"
Nov 03 02:56:15.503895 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:15.503874    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-metrics-certs\") pod \"maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx\" (UID: \"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"
Nov 03 02:56:15.555951 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:15.555879    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmcqs\" (UniqueName: \"kubernetes.io/projected/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-kube-api-access-tmcqs\") pod \"maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx\" (UID: \"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"
Nov 03 02:56:16.021885 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:16.021849    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-maas-api-tls\") pod \"maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx\" (UID: \"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"
Nov 03 02:56:16.024354 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:16.024333    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-maas-api-tls\") pod \"maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx\" (UID: \"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"
Nov 03 02:56:16.212040 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:16.212006    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"
Nov 03 02:56:16.584924 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:16.584847    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"]
Nov 03 02:56:16.590210 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:56:16.590178    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b69c55_a8e5_4fb9_8e61_e8533ee739a2.slice/crio-67b8d5c903513c4980e365ff15e46283b0afa1facb351237e34b2726ba148bf0 WatchSource:0}: Error finding container 67b8d5c903513c4980e365ff15e46283b0afa1facb351237e34b2726ba148bf0: Status 404 returned error can't find the container with id 67b8d5c903513c4980e365ff15e46283b0afa1facb351237e34b2726ba148bf0
Nov 03 02:56:17.023031 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:17.022990    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx" event={"ID":"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2","Type":"ContainerStarted","Data":"67b8d5c903513c4980e365ff15e46283b0afa1facb351237e34b2726ba148bf0"}
Nov 03 02:56:18.031490 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:18.031445    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx" event={"ID":"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2","Type":"ContainerStarted","Data":"d33788bc7f209a461a60ef36e631fa986eeea69fa0e5a7a8d89aeffbaa210dd5"}
Nov 03 02:56:18.031909 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:18.031567    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"
Nov 03 02:56:18.140671 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:18.140618    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx" podStartSLOduration=2.082368816 podStartE2EDuration="3.140604093s" totalImagesPullingTime="1.058235277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:56:18.138582255 +0000 UTC m=+3905.152823620" watchObservedRunningTime="2026-11-03 02:56:18.140604093 +0000 UTC m=+3905.154845458"
Nov 03 02:56:24.043143 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:24.043064    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"
Nov 03 02:56:30.714500 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:56:30.714463    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:56:49.253143 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:49.253100    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"]
Nov 03 02:56:49.258028 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:49.257999    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"
Nov 03 02:56:49.353209 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:49.353179    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ddcbd0-serving-cert\""
Nov 03 02:56:49.356809 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:49.356785    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"]
Nov 03 02:56:49.404603 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:49.404557    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2e9fe34-52c0-431a-8088-2783488fa3f0-maas-api-tls\") pod \"maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk\" (UID: \"d2e9fe34-52c0-431a-8088-2783488fa3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"
Nov 03 02:56:49.404806 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:49.404730    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztc86\" (UniqueName: \"kubernetes.io/projected/d2e9fe34-52c0-431a-8088-2783488fa3f0-kube-api-access-ztc86\") pod \"maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk\" (UID: \"d2e9fe34-52c0-431a-8088-2783488fa3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"
Nov 03 02:56:49.404806 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:49.404756    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e9fe34-52c0-431a-8088-2783488fa3f0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk\" (UID: \"d2e9fe34-52c0-431a-8088-2783488fa3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"
Nov 03 02:56:49.404950 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:49.404828    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2e9fe34-52c0-431a-8088-2783488fa3f0-metrics-certs\") pod \"maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk\" (UID: \"d2e9fe34-52c0-431a-8088-2783488fa3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"
Nov 03 02:56:49.506046 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:49.505964    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2e9fe34-52c0-431a-8088-2783488fa3f0-maas-api-tls\") pod \"maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk\" (UID: \"d2e9fe34-52c0-431a-8088-2783488fa3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"
Nov 03 02:56:49.506201 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:49.506050    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztc86\" (UniqueName: \"kubernetes.io/projected/d2e9fe34-52c0-431a-8088-2783488fa3f0-kube-api-access-ztc86\") pod \"maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk\" (UID: \"d2e9fe34-52c0-431a-8088-2783488fa3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"
Nov 03 02:56:49.506201 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:49.506071    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e9fe34-52c0-431a-8088-2783488fa3f0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk\" (UID: \"d2e9fe34-52c0-431a-8088-2783488fa3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"
Nov 03 02:56:49.506201 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:49.506123    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2e9fe34-52c0-431a-8088-2783488fa3f0-metrics-certs\") pod \"maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk\" (UID: \"d2e9fe34-52c0-431a-8088-2783488fa3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"
Nov 03 02:56:49.506867 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:49.506840    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e9fe34-52c0-431a-8088-2783488fa3f0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk\" (UID: \"d2e9fe34-52c0-431a-8088-2783488fa3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"
Nov 03 02:56:49.508664 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:49.508637    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2e9fe34-52c0-431a-8088-2783488fa3f0-metrics-certs\") pod \"maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk\" (UID: \"d2e9fe34-52c0-431a-8088-2783488fa3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"
Nov 03 02:56:49.508819 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:49.508796    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2e9fe34-52c0-431a-8088-2783488fa3f0-maas-api-tls\") pod \"maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk\" (UID: \"d2e9fe34-52c0-431a-8088-2783488fa3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"
Nov 03 02:56:49.559052 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:49.559020    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztc86\" (UniqueName: \"kubernetes.io/projected/d2e9fe34-52c0-431a-8088-2783488fa3f0-kube-api-access-ztc86\") pod \"maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk\" (UID: \"d2e9fe34-52c0-431a-8088-2783488fa3f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"
Nov 03 02:56:49.571849 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:49.571822    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"
Nov 03 02:56:49.945705 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:49.945620    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"]
Nov 03 02:56:49.951382 ip-10-0-139-26 kubenswrapper[5005]: W1103 02:56:49.951353    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e9fe34_52c0_431a_8088_2783488fa3f0.slice/crio-3f264ec19299d7c3143e7ba7a15552a9914c8b1d433982870ecaf1ab4aad17f5 WatchSource:0}: Error finding container 3f264ec19299d7c3143e7ba7a15552a9914c8b1d433982870ecaf1ab4aad17f5: Status 404 returned error can't find the container with id 3f264ec19299d7c3143e7ba7a15552a9914c8b1d433982870ecaf1ab4aad17f5
Nov 03 02:56:50.282203 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:50.282169    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk" event={"ID":"d2e9fe34-52c0-431a-8088-2783488fa3f0","Type":"ContainerStarted","Data":"3f264ec19299d7c3143e7ba7a15552a9914c8b1d433982870ecaf1ab4aad17f5"}
Nov 03 02:56:51.290304 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:51.290271    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk" event={"ID":"d2e9fe34-52c0-431a-8088-2783488fa3f0","Type":"ContainerStarted","Data":"bd3e2946120f981bc2f8d6210d3e6044e00a7d14aa19f7f80c2922758221013b"}
Nov 03 02:56:51.290765 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:51.290483    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"
Nov 03 02:56:51.403352 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:51.403291    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk" podStartSLOduration=1.51279589 podStartE2EDuration="2.403277905s" totalImagesPullingTime="890.482015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:56:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:56:51.401129948 +0000 UTC m=+3938.415371327" watchObservedRunningTime="2026-11-03 02:56:51.403277905 +0000 UTC m=+3938.417519284"
Nov 03 02:56:57.310318 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:56:57.310285    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"
Nov 03 02:57:00.715155 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:57:00.715114    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:57:30.714099 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:57:30.714060    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:57:31.754029 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:31.753997    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"]
Nov 03 02:57:31.754524 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:31.754290    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926" podUID="22dc475f-cfa4-435c-b021-ccdd2f2eec91" containerName="maas-api" containerID="cri-o://1b254586f73e1f779cc73fea05b98b5f9eea96775cbd9222f548ae39e3f8526b" gracePeriod=30
Nov 03 02:57:32.099826 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:32.099801    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"
Nov 03 02:57:32.204416 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:32.204379    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22dc475f-cfa4-435c-b021-ccdd2f2eec91-kube-api-access-q2s5c\" (UniqueName: \"kubernetes.io/projected/22dc475f-cfa4-435c-b021-ccdd2f2eec91-kube-api-access-q2s5c\") pod \"22dc475f-cfa4-435c-b021-ccdd2f2eec91\" (UID: \"22dc475f-cfa4-435c-b021-ccdd2f2eec91\") "
Nov 03 02:57:32.204631 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:32.204560    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22dc475f-cfa4-435c-b021-ccdd2f2eec91-metrics-certs\" (UniqueName: \"kubernetes.io/secret/22dc475f-cfa4-435c-b021-ccdd2f2eec91-metrics-certs\") pod \"22dc475f-cfa4-435c-b021-ccdd2f2eec91\" (UID: \"22dc475f-cfa4-435c-b021-ccdd2f2eec91\") "
Nov 03 02:57:32.204755 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:32.204635    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22dc475f-cfa4-435c-b021-ccdd2f2eec91-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22dc475f-cfa4-435c-b021-ccdd2f2eec91-openshift-service-ca\") pod \"22dc475f-cfa4-435c-b021-ccdd2f2eec91\" (UID: \"22dc475f-cfa4-435c-b021-ccdd2f2eec91\") "
Nov 03 02:57:32.204755 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:32.204729    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22dc475f-cfa4-435c-b021-ccdd2f2eec91-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22dc475f-cfa4-435c-b021-ccdd2f2eec91-maas-api-tls\") pod \"22dc475f-cfa4-435c-b021-ccdd2f2eec91\" (UID: \"22dc475f-cfa4-435c-b021-ccdd2f2eec91\") "
Nov 03 02:57:32.207524 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:32.207487    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22dc475f-cfa4-435c-b021-ccdd2f2eec91-openshift-service-ca" pod "22dc475f-cfa4-435c-b021-ccdd2f2eec91" (UID: "22dc475f-cfa4-435c-b021-ccdd2f2eec91"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:57:32.207969 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:32.207946    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22dc475f-cfa4-435c-b021-ccdd2f2eec91-metrics-certs" pod "22dc475f-cfa4-435c-b021-ccdd2f2eec91" (UID: "22dc475f-cfa4-435c-b021-ccdd2f2eec91"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:57:32.208431 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:32.208407    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22dc475f-cfa4-435c-b021-ccdd2f2eec91-maas-api-tls" pod "22dc475f-cfa4-435c-b021-ccdd2f2eec91" (UID: "22dc475f-cfa4-435c-b021-ccdd2f2eec91"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:57:32.208951 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:32.208924    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22dc475f-cfa4-435c-b021-ccdd2f2eec91-kube-api-access-q2s5c" pod "22dc475f-cfa4-435c-b021-ccdd2f2eec91" (UID: "22dc475f-cfa4-435c-b021-ccdd2f2eec91"). InnerVolumeSpecName "kube-api-access-q2s5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:57:32.306481 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:32.306446    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22dc475f-cfa4-435c-b021-ccdd2f2eec91-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:57:32.306481 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:32.306480    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22dc475f-cfa4-435c-b021-ccdd2f2eec91-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:57:32.306683 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:32.306493    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22dc475f-cfa4-435c-b021-ccdd2f2eec91-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:57:32.306683 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:32.306507    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2s5c\" (UniqueName: \"kubernetes.io/projected/22dc475f-cfa4-435c-b021-ccdd2f2eec91-kube-api-access-q2s5c\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:57:32.582652 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:32.582618    5005 generic.go:356] "Generic (PLEG): container finished" podID="22dc475f-cfa4-435c-b021-ccdd2f2eec91" containerID="1b254586f73e1f779cc73fea05b98b5f9eea96775cbd9222f548ae39e3f8526b" exitCode=0
Nov 03 02:57:32.582854 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:32.582670    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"
Nov 03 02:57:32.582854 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:32.582720    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926" event={"ID":"22dc475f-cfa4-435c-b021-ccdd2f2eec91","Type":"ContainerDied","Data":"1b254586f73e1f779cc73fea05b98b5f9eea96775cbd9222f548ae39e3f8526b"}
Nov 03 02:57:32.582854 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:32.582777    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926" event={"ID":"22dc475f-cfa4-435c-b021-ccdd2f2eec91","Type":"ContainerDied","Data":"c6487e6331f1b1230f1e7b72e076c3d39417996b3f0e6197d079d0b4167d2281"}
Nov 03 02:57:32.582854 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:32.582800    5005 scope.go:122] "RemoveContainer" containerID="1b254586f73e1f779cc73fea05b98b5f9eea96775cbd9222f548ae39e3f8526b"
Nov 03 02:57:32.594057 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:32.594041    5005 scope.go:122] "RemoveContainer" containerID="1b254586f73e1f779cc73fea05b98b5f9eea96775cbd9222f548ae39e3f8526b"
Nov 03 02:57:32.594313 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:57:32.594296    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b254586f73e1f779cc73fea05b98b5f9eea96775cbd9222f548ae39e3f8526b\": container with ID starting with 1b254586f73e1f779cc73fea05b98b5f9eea96775cbd9222f548ae39e3f8526b not found: ID does not exist" containerID="1b254586f73e1f779cc73fea05b98b5f9eea96775cbd9222f548ae39e3f8526b"
Nov 03 02:57:32.594359 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:32.594322    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b254586f73e1f779cc73fea05b98b5f9eea96775cbd9222f548ae39e3f8526b"} err="failed to get container status \"1b254586f73e1f779cc73fea05b98b5f9eea96775cbd9222f548ae39e3f8526b\": rpc error: code = NotFound desc = could not find container \"1b254586f73e1f779cc73fea05b98b5f9eea96775cbd9222f548ae39e3f8526b\": container with ID starting with 1b254586f73e1f779cc73fea05b98b5f9eea96775cbd9222f548ae39e3f8526b not found: ID does not exist"
Nov 03 02:57:32.754791 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:32.754742    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"]
Nov 03 02:57:32.803186 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:32.803151    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e039cb-846897867f-rs926"]
Nov 03 02:57:33.110006 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:33.109958    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22dc475f-cfa4-435c-b021-ccdd2f2eec91" path="/var/lib/kubelet/pods/22dc475f-cfa4-435c-b021-ccdd2f2eec91/volumes"
Nov 03 02:57:52.782062 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:52.781858    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"]
Nov 03 02:57:52.782643 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:52.782113    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx" podUID="a0b69c55-a8e5-4fb9-8e61-e8533ee739a2" containerName="maas-api" containerID="cri-o://d33788bc7f209a461a60ef36e631fa986eeea69fa0e5a7a8d89aeffbaa210dd5" gracePeriod=30
Nov 03 02:57:53.125945 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:53.125917    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"
Nov 03 02:57:53.229199 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:53.229145    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-openshift-service-ca\") pod \"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2\" (UID: \"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2\") "
Nov 03 02:57:53.229199 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:53.229208    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-maas-api-tls\") pod \"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2\" (UID: \"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2\") "
Nov 03 02:57:53.229456 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:53.229233    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-kube-api-access-tmcqs\" (UniqueName: \"kubernetes.io/projected/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-kube-api-access-tmcqs\") pod \"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2\" (UID: \"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2\") "
Nov 03 02:57:53.229456 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:53.229316    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-metrics-certs\") pod \"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2\" (UID: \"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2\") "
Nov 03 02:57:53.229582 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:53.229492    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-openshift-service-ca" pod "a0b69c55-a8e5-4fb9-8e61-e8533ee739a2" (UID: "a0b69c55-a8e5-4fb9-8e61-e8533ee739a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:57:53.229751 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:53.229723    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:57:53.231452 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:53.231423    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-kube-api-access-tmcqs" pod "a0b69c55-a8e5-4fb9-8e61-e8533ee739a2" (UID: "a0b69c55-a8e5-4fb9-8e61-e8533ee739a2"). InnerVolumeSpecName "kube-api-access-tmcqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:57:53.231535 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:53.231429    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-metrics-certs" pod "a0b69c55-a8e5-4fb9-8e61-e8533ee739a2" (UID: "a0b69c55-a8e5-4fb9-8e61-e8533ee739a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:57:53.231535 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:53.231461    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-maas-api-tls" pod "a0b69c55-a8e5-4fb9-8e61-e8533ee739a2" (UID: "a0b69c55-a8e5-4fb9-8e61-e8533ee739a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:57:53.331753 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:53.331652    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:57:53.331753 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:53.331689    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:57:53.331753 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:53.331724    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmcqs\" (UniqueName: \"kubernetes.io/projected/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2-kube-api-access-tmcqs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:57:53.723186 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:53.723153    5005 generic.go:356] "Generic (PLEG): container finished" podID="a0b69c55-a8e5-4fb9-8e61-e8533ee739a2" containerID="d33788bc7f209a461a60ef36e631fa986eeea69fa0e5a7a8d89aeffbaa210dd5" exitCode=0
Nov 03 02:57:53.723379 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:53.723211    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"
Nov 03 02:57:53.723379 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:53.723245    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx" event={"ID":"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2","Type":"ContainerDied","Data":"d33788bc7f209a461a60ef36e631fa986eeea69fa0e5a7a8d89aeffbaa210dd5"}
Nov 03 02:57:53.723379 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:53.723325    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx" event={"ID":"a0b69c55-a8e5-4fb9-8e61-e8533ee739a2","Type":"ContainerDied","Data":"67b8d5c903513c4980e365ff15e46283b0afa1facb351237e34b2726ba148bf0"}
Nov 03 02:57:53.723379 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:53.723354    5005 scope.go:122] "RemoveContainer" containerID="d33788bc7f209a461a60ef36e631fa986eeea69fa0e5a7a8d89aeffbaa210dd5"
Nov 03 02:57:53.786717 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:53.785057    5005 scope.go:122] "RemoveContainer" containerID="d33788bc7f209a461a60ef36e631fa986eeea69fa0e5a7a8d89aeffbaa210dd5"
Nov 03 02:57:53.799714 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:57:53.789100    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d33788bc7f209a461a60ef36e631fa986eeea69fa0e5a7a8d89aeffbaa210dd5\": container with ID starting with d33788bc7f209a461a60ef36e631fa986eeea69fa0e5a7a8d89aeffbaa210dd5 not found: ID does not exist" containerID="d33788bc7f209a461a60ef36e631fa986eeea69fa0e5a7a8d89aeffbaa210dd5"
Nov 03 02:57:53.799714 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:53.789150    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d33788bc7f209a461a60ef36e631fa986eeea69fa0e5a7a8d89aeffbaa210dd5"} err="failed to get container status \"d33788bc7f209a461a60ef36e631fa986eeea69fa0e5a7a8d89aeffbaa210dd5\": rpc error: code = NotFound desc = could not find container \"d33788bc7f209a461a60ef36e631fa986eeea69fa0e5a7a8d89aeffbaa210dd5\": container with ID starting with d33788bc7f209a461a60ef36e631fa986eeea69fa0e5a7a8d89aeffbaa210dd5 not found: ID does not exist"
Nov 03 02:57:53.901766 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:53.901728    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"]
Nov 03 02:57:53.947745 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:53.947684    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c43e9-6c5cd5cdf4-2mqfx"]
Nov 03 02:57:55.111040 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:57:55.111008    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0b69c55-a8e5-4fb9-8e61-e8533ee739a2" path="/var/lib/kubelet/pods/a0b69c55-a8e5-4fb9-8e61-e8533ee739a2/volumes"
Nov 03 02:58:00.715585 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:58:00.715546    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:58:30.715470 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:58:30.715430    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:59:00.719772 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:59:00.719683    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:59:30.781720 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:59:30.781484    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 02:59:54.447814 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:54.447777    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"]
Nov 03 02:59:54.448485 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:54.447966    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9" podUID="f2b37f9f-b995-403b-8ba8-e27273057f9f" containerName="maas-api" containerID="cri-o://ec06bd451c3194bbade2e1e3088afc24fb8839fb13747b2e3c4d411f2d576bec" gracePeriod=30
Nov 03 02:59:54.768301 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:54.768275    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"
Nov 03 02:59:54.808152 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:54.808122    5005 generic.go:356] "Generic (PLEG): container finished" podID="f2b37f9f-b995-403b-8ba8-e27273057f9f" containerID="ec06bd451c3194bbade2e1e3088afc24fb8839fb13747b2e3c4d411f2d576bec" exitCode=0
Nov 03 02:59:54.808296 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:54.808170    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"
Nov 03 02:59:54.808296 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:54.808207    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9" event={"ID":"f2b37f9f-b995-403b-8ba8-e27273057f9f","Type":"ContainerDied","Data":"ec06bd451c3194bbade2e1e3088afc24fb8839fb13747b2e3c4d411f2d576bec"}
Nov 03 02:59:54.808296 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:54.808263    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9" event={"ID":"f2b37f9f-b995-403b-8ba8-e27273057f9f","Type":"ContainerDied","Data":"95d12329a596dedfc94b27c2b52f7e62c2be541c83e5a086ad53c68c8d3dc9bc"}
Nov 03 02:59:54.808296 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:54.808291    5005 scope.go:122] "RemoveContainer" containerID="ec06bd451c3194bbade2e1e3088afc24fb8839fb13747b2e3c4d411f2d576bec"
Nov 03 02:59:54.816806 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:54.816781    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b37f9f-b995-403b-8ba8-e27273057f9f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b37f9f-b995-403b-8ba8-e27273057f9f-metrics-certs\") pod \"f2b37f9f-b995-403b-8ba8-e27273057f9f\" (UID: \"f2b37f9f-b995-403b-8ba8-e27273057f9f\") "
Nov 03 02:59:54.816935 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:54.816878    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b37f9f-b995-403b-8ba8-e27273057f9f-kube-api-access-v8pxc\" (UniqueName: \"kubernetes.io/projected/f2b37f9f-b995-403b-8ba8-e27273057f9f-kube-api-access-v8pxc\") pod \"f2b37f9f-b995-403b-8ba8-e27273057f9f\" (UID: \"f2b37f9f-b995-403b-8ba8-e27273057f9f\") "
Nov 03 02:59:54.816935 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:54.816917    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b37f9f-b995-403b-8ba8-e27273057f9f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b37f9f-b995-403b-8ba8-e27273057f9f-openshift-service-ca\") pod \"f2b37f9f-b995-403b-8ba8-e27273057f9f\" (UID: \"f2b37f9f-b995-403b-8ba8-e27273057f9f\") "
Nov 03 02:59:54.817049 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:54.816954    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b37f9f-b995-403b-8ba8-e27273057f9f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b37f9f-b995-403b-8ba8-e27273057f9f-maas-api-tls\") pod \"f2b37f9f-b995-403b-8ba8-e27273057f9f\" (UID: \"f2b37f9f-b995-403b-8ba8-e27273057f9f\") "
Nov 03 02:59:54.817452 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:54.817422    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b37f9f-b995-403b-8ba8-e27273057f9f-openshift-service-ca" pod "f2b37f9f-b995-403b-8ba8-e27273057f9f" (UID: "f2b37f9f-b995-403b-8ba8-e27273057f9f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:59:54.818529 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:54.818507    5005 scope.go:122] "RemoveContainer" containerID="ec06bd451c3194bbade2e1e3088afc24fb8839fb13747b2e3c4d411f2d576bec"
Nov 03 02:59:54.818897 ip-10-0-139-26 kubenswrapper[5005]: E1103 02:59:54.818873    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec06bd451c3194bbade2e1e3088afc24fb8839fb13747b2e3c4d411f2d576bec\": container with ID starting with ec06bd451c3194bbade2e1e3088afc24fb8839fb13747b2e3c4d411f2d576bec not found: ID does not exist" containerID="ec06bd451c3194bbade2e1e3088afc24fb8839fb13747b2e3c4d411f2d576bec"
Nov 03 02:59:54.819009 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:54.818915    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec06bd451c3194bbade2e1e3088afc24fb8839fb13747b2e3c4d411f2d576bec"} err="failed to get container status \"ec06bd451c3194bbade2e1e3088afc24fb8839fb13747b2e3c4d411f2d576bec\": rpc error: code = NotFound desc = could not find container \"ec06bd451c3194bbade2e1e3088afc24fb8839fb13747b2e3c4d411f2d576bec\": container with ID starting with ec06bd451c3194bbade2e1e3088afc24fb8839fb13747b2e3c4d411f2d576bec not found: ID does not exist"
Nov 03 02:59:54.819118 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:54.819096    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b37f9f-b995-403b-8ba8-e27273057f9f-metrics-certs" pod "f2b37f9f-b995-403b-8ba8-e27273057f9f" (UID: "f2b37f9f-b995-403b-8ba8-e27273057f9f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:59:54.819480 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:54.819456    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b37f9f-b995-403b-8ba8-e27273057f9f-maas-api-tls" pod "f2b37f9f-b995-403b-8ba8-e27273057f9f" (UID: "f2b37f9f-b995-403b-8ba8-e27273057f9f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:59:54.819836 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:54.819818    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b37f9f-b995-403b-8ba8-e27273057f9f-kube-api-access-v8pxc" pod "f2b37f9f-b995-403b-8ba8-e27273057f9f" (UID: "f2b37f9f-b995-403b-8ba8-e27273057f9f"). InnerVolumeSpecName "kube-api-access-v8pxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:59:54.919205 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:54.919170    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b37f9f-b995-403b-8ba8-e27273057f9f-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:59:54.919434 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:54.919206    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8pxc\" (UniqueName: \"kubernetes.io/projected/f2b37f9f-b995-403b-8ba8-e27273057f9f-kube-api-access-v8pxc\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:59:54.919434 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:54.919224    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b37f9f-b995-403b-8ba8-e27273057f9f-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:59:54.919434 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:54.919239    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b37f9f-b995-403b-8ba8-e27273057f9f-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:59:55.234959 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:55.234901    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"]
Nov 03 02:59:55.237986 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:55.237960    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ecdbc7-748b9f94b9-zrjt9"]
Nov 03 02:59:57.110285 ip-10-0-139-26 kubenswrapper[5005]: I1103 02:59:57.110248    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b37f9f-b995-403b-8ba8-e27273057f9f" path="/var/lib/kubelet/pods/f2b37f9f-b995-403b-8ba8-e27273057f9f/volumes"
Nov 03 03:00:00.717338 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:00:00.717300    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:00:30.713520 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:00:30.713475    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:00:42.009303 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:42.009266    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"]
Nov 03 03:00:42.009839 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:42.009491    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv" podUID="b80d8879-30b7-477c-8b8e-5c806d813e54" containerName="istio-proxy" containerID="cri-o://c45cb38dc33257fbd4aa89532221683af3b5ca4bb856c3d257f1610dd5f000dd" gracePeriod=30
Nov 03 03:00:42.059002 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:42.058642    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"]
Nov 03 03:00:42.059627 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:42.059571    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc" podUID="6a37751f-fd21-4160-a44f-e97a796f9198" containerName="istio-proxy" containerID="cri-o://8f80e6ed396306f9351ec492c37fd65594d0a20d880be87e108ef96185069a8a" gracePeriod=30
Nov 03 03:00:47.201744 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.201261    5005 generic.go:356] "Generic (PLEG): container finished" podID="b80d8879-30b7-477c-8b8e-5c806d813e54" containerID="c45cb38dc33257fbd4aa89532221683af3b5ca4bb856c3d257f1610dd5f000dd" exitCode=0
Nov 03 03:00:47.201744 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.201336    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv" event={"ID":"b80d8879-30b7-477c-8b8e-5c806d813e54","Type":"ContainerDied","Data":"c45cb38dc33257fbd4aa89532221683af3b5ca4bb856c3d257f1610dd5f000dd"}
Nov 03 03:00:47.295743 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.294843    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 03:00:47.369605 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.369572    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-credential-socket\") pod \"b80d8879-30b7-477c-8b8e-5c806d813e54\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") "
Nov 03 03:00:47.369760 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.369653    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-workload-certs\") pod \"b80d8879-30b7-477c-8b8e-5c806d813e54\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") "
Nov 03 03:00:47.369760 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.369674    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-data\") pod \"b80d8879-30b7-477c-8b8e-5c806d813e54\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") "
Nov 03 03:00:47.369760 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.369729    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b80d8879-30b7-477c-8b8e-5c806d813e54-kube-api-access-gcfms\" (UniqueName: \"kubernetes.io/projected/b80d8879-30b7-477c-8b8e-5c806d813e54-kube-api-access-gcfms\") pod \"b80d8879-30b7-477c-8b8e-5c806d813e54\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") "
Nov 03 03:00:47.369942 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.369768    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-envoy\") pod \"b80d8879-30b7-477c-8b8e-5c806d813e54\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") "
Nov 03 03:00:47.369942 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.369867    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-token\" (UniqueName: \"kubernetes.io/projected/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-token\") pod \"b80d8879-30b7-477c-8b8e-5c806d813e54\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") "
Nov 03 03:00:47.369942 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.369928    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-credential-socket" pod "b80d8879-30b7-477c-8b8e-5c806d813e54" (UID: "b80d8879-30b7-477c-8b8e-5c806d813e54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:00:47.370109 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.369945    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b80d8879-30b7-477c-8b8e-5c806d813e54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b80d8879-30b7-477c-8b8e-5c806d813e54-istiod-ca-cert\") pod \"b80d8879-30b7-477c-8b8e-5c806d813e54\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") "
Nov 03 03:00:47.370109 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.369985    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-workload-socket\") pod \"b80d8879-30b7-477c-8b8e-5c806d813e54\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") "
Nov 03 03:00:47.370109 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.370006    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-podinfo\") pod \"b80d8879-30b7-477c-8b8e-5c806d813e54\" (UID: \"b80d8879-30b7-477c-8b8e-5c806d813e54\") "
Nov 03 03:00:47.370109 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.370031    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-data" pod "b80d8879-30b7-477c-8b8e-5c806d813e54" (UID: "b80d8879-30b7-477c-8b8e-5c806d813e54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:00:47.370323 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.370252    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-workload-socket" pod "b80d8879-30b7-477c-8b8e-5c806d813e54" (UID: "b80d8879-30b7-477c-8b8e-5c806d813e54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:00:47.370323 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.370303    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-workload-certs" pod "b80d8879-30b7-477c-8b8e-5c806d813e54" (UID: "b80d8879-30b7-477c-8b8e-5c806d813e54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:00:47.370437 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.370356    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b80d8879-30b7-477c-8b8e-5c806d813e54-istiod-ca-cert" pod "b80d8879-30b7-477c-8b8e-5c806d813e54" (UID: "b80d8879-30b7-477c-8b8e-5c806d813e54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:00:47.370671 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.370623    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:47.370671 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.370644    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:47.370671 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.370657    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b80d8879-30b7-477c-8b8e-5c806d813e54-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:47.370671 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.370670    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:47.370877 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.370683    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:47.372109 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.372061    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-envoy" pod "b80d8879-30b7-477c-8b8e-5c806d813e54" (UID: "b80d8879-30b7-477c-8b8e-5c806d813e54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:00:47.372624 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.372599    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-token" pod "b80d8879-30b7-477c-8b8e-5c806d813e54" (UID: "b80d8879-30b7-477c-8b8e-5c806d813e54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:00:47.372731 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.372607    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-podinfo" pod "b80d8879-30b7-477c-8b8e-5c806d813e54" (UID: "b80d8879-30b7-477c-8b8e-5c806d813e54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:00:47.372731 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.372607    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b80d8879-30b7-477c-8b8e-5c806d813e54-kube-api-access-gcfms" pod "b80d8879-30b7-477c-8b8e-5c806d813e54" (UID: "b80d8879-30b7-477c-8b8e-5c806d813e54"). InnerVolumeSpecName "kube-api-access-gcfms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:00:47.379164 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.379144    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 03:00:47.471954 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.471918    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-credential-socket\") pod \"6a37751f-fd21-4160-a44f-e97a796f9198\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") "
Nov 03 03:00:47.472256 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.471970    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a37751f-fd21-4160-a44f-e97a796f9198-istio-token\" (UniqueName: \"kubernetes.io/projected/6a37751f-fd21-4160-a44f-e97a796f9198-istio-token\") pod \"6a37751f-fd21-4160-a44f-e97a796f9198\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") "
Nov 03 03:00:47.472256 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.471990    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a37751f-fd21-4160-a44f-e97a796f9198-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a37751f-fd21-4160-a44f-e97a796f9198-istiod-ca-cert\") pod \"6a37751f-fd21-4160-a44f-e97a796f9198\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") "
Nov 03 03:00:47.472256 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.472016    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-workload-certs\") pod \"6a37751f-fd21-4160-a44f-e97a796f9198\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") "
Nov 03 03:00:47.472256 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.472067    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a37751f-fd21-4160-a44f-e97a796f9198-kube-api-access-gwnz6\" (UniqueName: \"kubernetes.io/projected/6a37751f-fd21-4160-a44f-e97a796f9198-kube-api-access-gwnz6\") pod \"6a37751f-fd21-4160-a44f-e97a796f9198\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") "
Nov 03 03:00:47.472256 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.472137    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-istio-envoy\") pod \"6a37751f-fd21-4160-a44f-e97a796f9198\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") "
Nov 03 03:00:47.472256 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.472226    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-workload-socket\") pod \"6a37751f-fd21-4160-a44f-e97a796f9198\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") "
Nov 03 03:00:47.472634 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.472270    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-credential-socket" pod "6a37751f-fd21-4160-a44f-e97a796f9198" (UID: "6a37751f-fd21-4160-a44f-e97a796f9198"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:00:47.472634 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.472314    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a37751f-fd21-4160-a44f-e97a796f9198-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a37751f-fd21-4160-a44f-e97a796f9198-istio-podinfo\") pod \"6a37751f-fd21-4160-a44f-e97a796f9198\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") "
Nov 03 03:00:47.472634 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.472400    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-istio-data\") pod \"6a37751f-fd21-4160-a44f-e97a796f9198\" (UID: \"6a37751f-fd21-4160-a44f-e97a796f9198\") "
Nov 03 03:00:47.472634 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.472442    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-workload-certs" pod "6a37751f-fd21-4160-a44f-e97a796f9198" (UID: "6a37751f-fd21-4160-a44f-e97a796f9198"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:00:47.472634 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.472545    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a37751f-fd21-4160-a44f-e97a796f9198-istiod-ca-cert" pod "6a37751f-fd21-4160-a44f-e97a796f9198" (UID: "6a37751f-fd21-4160-a44f-e97a796f9198"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:00:47.472922 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.472670    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-workload-socket" pod "6a37751f-fd21-4160-a44f-e97a796f9198" (UID: "6a37751f-fd21-4160-a44f-e97a796f9198"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:00:47.472922 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.472872    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-istio-data" pod "6a37751f-fd21-4160-a44f-e97a796f9198" (UID: "6a37751f-fd21-4160-a44f-e97a796f9198"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:00:47.473173 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.473030    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:47.473173 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.473054    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcfms\" (UniqueName: \"kubernetes.io/projected/b80d8879-30b7-477c-8b8e-5c806d813e54-kube-api-access-gcfms\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:47.473173 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.473069    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:47.473173 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.473081    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:47.473173 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.473094    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:47.473173 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.473107    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:47.473173 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.473121    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a37751f-fd21-4160-a44f-e97a796f9198-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:47.473173 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.473135    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:47.473173 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.473148    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b80d8879-30b7-477c-8b8e-5c806d813e54-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:47.474972 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.474935    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a37751f-fd21-4160-a44f-e97a796f9198-istio-podinfo" pod "6a37751f-fd21-4160-a44f-e97a796f9198" (UID: "6a37751f-fd21-4160-a44f-e97a796f9198"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:00:47.475254 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.475230    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-istio-envoy" pod "6a37751f-fd21-4160-a44f-e97a796f9198" (UID: "6a37751f-fd21-4160-a44f-e97a796f9198"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:00:47.475338 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.475271    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a37751f-fd21-4160-a44f-e97a796f9198-istio-token" pod "6a37751f-fd21-4160-a44f-e97a796f9198" (UID: "6a37751f-fd21-4160-a44f-e97a796f9198"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:00:47.475338 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.475302    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a37751f-fd21-4160-a44f-e97a796f9198-kube-api-access-gwnz6" pod "6a37751f-fd21-4160-a44f-e97a796f9198" (UID: "6a37751f-fd21-4160-a44f-e97a796f9198"). InnerVolumeSpecName "kube-api-access-gwnz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:00:47.574829 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.574745    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a37751f-fd21-4160-a44f-e97a796f9198-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:47.574829 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.574776    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a37751f-fd21-4160-a44f-e97a796f9198-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:47.574829 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.574790    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwnz6\" (UniqueName: \"kubernetes.io/projected/6a37751f-fd21-4160-a44f-e97a796f9198-kube-api-access-gwnz6\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:47.574829 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:47.574805    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a37751f-fd21-4160-a44f-e97a796f9198-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:48.209038 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:48.208999    5005 generic.go:356] "Generic (PLEG): container finished" podID="6a37751f-fd21-4160-a44f-e97a796f9198" containerID="8f80e6ed396306f9351ec492c37fd65594d0a20d880be87e108ef96185069a8a" exitCode=0
Nov 03 03:00:48.209467 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:48.209050    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"
Nov 03 03:00:48.209467 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:48.209073    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc" event={"ID":"6a37751f-fd21-4160-a44f-e97a796f9198","Type":"ContainerDied","Data":"8f80e6ed396306f9351ec492c37fd65594d0a20d880be87e108ef96185069a8a"}
Nov 03 03:00:48.209467 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:48.209128    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc" event={"ID":"6a37751f-fd21-4160-a44f-e97a796f9198","Type":"ContainerDied","Data":"b73f948d6fe1315069962d93b3cc693311c1f54a58bab9917cb4d9baa36e125e"}
Nov 03 03:00:48.209467 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:48.209151    5005 scope.go:122] "RemoveContainer" containerID="8f80e6ed396306f9351ec492c37fd65594d0a20d880be87e108ef96185069a8a"
Nov 03 03:00:48.211041 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:48.211006    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"
Nov 03 03:00:48.211164 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:48.211007    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv" event={"ID":"b80d8879-30b7-477c-8b8e-5c806d813e54","Type":"ContainerDied","Data":"f76af134dac86d2adaf02e8a3a282c6c56b797f462011539a6cc8b672bc71bc4"}
Nov 03 03:00:48.225014 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:48.224976    5005 scope.go:122] "RemoveContainer" containerID="8f80e6ed396306f9351ec492c37fd65594d0a20d880be87e108ef96185069a8a"
Nov 03 03:00:48.225423 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:00:48.225257    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f80e6ed396306f9351ec492c37fd65594d0a20d880be87e108ef96185069a8a\": container with ID starting with 8f80e6ed396306f9351ec492c37fd65594d0a20d880be87e108ef96185069a8a not found: ID does not exist" containerID="8f80e6ed396306f9351ec492c37fd65594d0a20d880be87e108ef96185069a8a"
Nov 03 03:00:48.225423 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:48.225287    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f80e6ed396306f9351ec492c37fd65594d0a20d880be87e108ef96185069a8a"} err="failed to get container status \"8f80e6ed396306f9351ec492c37fd65594d0a20d880be87e108ef96185069a8a\": rpc error: code = NotFound desc = could not find container \"8f80e6ed396306f9351ec492c37fd65594d0a20d880be87e108ef96185069a8a\": container with ID starting with 8f80e6ed396306f9351ec492c37fd65594d0a20d880be87e108ef96185069a8a not found: ID does not exist"
Nov 03 03:00:48.225423 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:48.225310    5005 scope.go:122] "RemoveContainer" containerID="c45cb38dc33257fbd4aa89532221683af3b5ca4bb856c3d257f1610dd5f000dd"
Nov 03 03:00:48.364463 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:48.364394    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"]
Nov 03 03:00:48.410710 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:48.410675    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-92npv"]
Nov 03 03:00:48.474569 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:48.474528    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"]
Nov 03 03:00:48.478288 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:48.478265    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-dc14f6-openshift-default-8848578fc-hkwpc"]
Nov 03 03:00:49.110456 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:49.110424    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a37751f-fd21-4160-a44f-e97a796f9198" path="/var/lib/kubelet/pods/6a37751f-fd21-4160-a44f-e97a796f9198/volumes"
Nov 03 03:00:49.110950 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:00:49.110926    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b80d8879-30b7-477c-8b8e-5c806d813e54" path="/var/lib/kubelet/pods/b80d8879-30b7-477c-8b8e-5c806d813e54/volumes"
Nov 03 03:01:00.714427 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:01:00.714341    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:01:10.253428 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:10.253383    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"]
Nov 03 03:01:10.254076 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:10.253646    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk" podUID="d2e9fe34-52c0-431a-8088-2783488fa3f0" containerName="maas-api" containerID="cri-o://bd3e2946120f981bc2f8d6210d3e6044e00a7d14aa19f7f80c2922758221013b" gracePeriod=30
Nov 03 03:01:10.394600 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:10.394560    5005 generic.go:356] "Generic (PLEG): container finished" podID="d2e9fe34-52c0-431a-8088-2783488fa3f0" containerID="bd3e2946120f981bc2f8d6210d3e6044e00a7d14aa19f7f80c2922758221013b" exitCode=0
Nov 03 03:01:10.394790 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:10.394630    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk" event={"ID":"d2e9fe34-52c0-431a-8088-2783488fa3f0","Type":"ContainerDied","Data":"bd3e2946120f981bc2f8d6210d3e6044e00a7d14aa19f7f80c2922758221013b"}
Nov 03 03:01:10.577996 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:10.577970    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"
Nov 03 03:01:10.606051 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:10.606019    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2e9fe34-52c0-431a-8088-2783488fa3f0-kube-api-access-ztc86\" (UniqueName: \"kubernetes.io/projected/d2e9fe34-52c0-431a-8088-2783488fa3f0-kube-api-access-ztc86\") pod \"d2e9fe34-52c0-431a-8088-2783488fa3f0\" (UID: \"d2e9fe34-52c0-431a-8088-2783488fa3f0\") "
Nov 03 03:01:10.606339 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:10.606316    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2e9fe34-52c0-431a-8088-2783488fa3f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2e9fe34-52c0-431a-8088-2783488fa3f0-metrics-certs\") pod \"d2e9fe34-52c0-431a-8088-2783488fa3f0\" (UID: \"d2e9fe34-52c0-431a-8088-2783488fa3f0\") "
Nov 03 03:01:10.606403 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:10.606359    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2e9fe34-52c0-431a-8088-2783488fa3f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e9fe34-52c0-431a-8088-2783488fa3f0-openshift-service-ca\") pod \"d2e9fe34-52c0-431a-8088-2783488fa3f0\" (UID: \"d2e9fe34-52c0-431a-8088-2783488fa3f0\") "
Nov 03 03:01:10.606463 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:10.606414    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2e9fe34-52c0-431a-8088-2783488fa3f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2e9fe34-52c0-431a-8088-2783488fa3f0-maas-api-tls\") pod \"d2e9fe34-52c0-431a-8088-2783488fa3f0\" (UID: \"d2e9fe34-52c0-431a-8088-2783488fa3f0\") "
Nov 03 03:01:10.606841 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:10.606811    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2e9fe34-52c0-431a-8088-2783488fa3f0-openshift-service-ca" pod "d2e9fe34-52c0-431a-8088-2783488fa3f0" (UID: "d2e9fe34-52c0-431a-8088-2783488fa3f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:01:10.608487 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:10.608462    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2e9fe34-52c0-431a-8088-2783488fa3f0-maas-api-tls" pod "d2e9fe34-52c0-431a-8088-2783488fa3f0" (UID: "d2e9fe34-52c0-431a-8088-2783488fa3f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:01:10.608729 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:10.608707    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2e9fe34-52c0-431a-8088-2783488fa3f0-metrics-certs" pod "d2e9fe34-52c0-431a-8088-2783488fa3f0" (UID: "d2e9fe34-52c0-431a-8088-2783488fa3f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:01:10.608787 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:10.608744    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e9fe34-52c0-431a-8088-2783488fa3f0-kube-api-access-ztc86" pod "d2e9fe34-52c0-431a-8088-2783488fa3f0" (UID: "d2e9fe34-52c0-431a-8088-2783488fa3f0"). InnerVolumeSpecName "kube-api-access-ztc86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:01:10.720749 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:10.717877    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2e9fe34-52c0-431a-8088-2783488fa3f0-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:01:10.720749 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:10.717921    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e9fe34-52c0-431a-8088-2783488fa3f0-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:01:10.720749 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:10.717935    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2e9fe34-52c0-431a-8088-2783488fa3f0-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:01:10.720749 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:10.717949    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztc86\" (UniqueName: \"kubernetes.io/projected/d2e9fe34-52c0-431a-8088-2783488fa3f0-kube-api-access-ztc86\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:01:11.403101 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:11.403023    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"
Nov 03 03:01:11.403101 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:11.403053    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk" event={"ID":"d2e9fe34-52c0-431a-8088-2783488fa3f0","Type":"ContainerDied","Data":"3f264ec19299d7c3143e7ba7a15552a9914c8b1d433982870ecaf1ab4aad17f5"}
Nov 03 03:01:11.403610 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:11.403118    5005 scope.go:122] "RemoveContainer" containerID="bd3e2946120f981bc2f8d6210d3e6044e00a7d14aa19f7f80c2922758221013b"
Nov 03 03:01:11.565590 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:11.565545    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"]
Nov 03 03:01:11.612010 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:11.611965    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ddcbd0-69f987cfd7-8vcrk"]
Nov 03 03:01:13.111579 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:13.111545    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2e9fe34-52c0-431a-8088-2783488fa3f0" path="/var/lib/kubelet/pods/d2e9fe34-52c0-431a-8088-2783488fa3f0/volumes"
Nov 03 03:01:13.724032 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:13.723913    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 03:01:13.739708 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:01:13.739666    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 03:01:30.715157 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:01:30.715109    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:02:00.714555 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:02:00.714517    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:02:30.713557 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:02:30.713476    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:03:00.714903 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:03:00.714856    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:03:30.714231 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:03:30.714185    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:04:00.718610 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:04:00.718530    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:04:30.714892 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:04:30.714854    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:05:00.713733 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:05:00.713670    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:05:30.714294 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:05:30.714215    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:06:00.714164 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:06:00.714127    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:06:13.768104 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:06:13.768077    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 03:06:13.796058 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:06:13.796033    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 03:06:30.713935 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:06:30.713899    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:07:00.718530 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:07:00.718428    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:07:30.715738 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:07:30.715686    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:08:00.185990 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:00.185954    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"]
Nov 03 03:08:00.186582 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:00.186222    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw" podUID="eded8d0f-e8e1-4410-baa9-e2982338f3e3" containerName="maas-api" containerID="cri-o://846eea1f15d4b0fb2c1dfffac87d0646576570dbfe329d5c36642f8de3893189" gracePeriod=30
Nov 03 03:08:00.514632 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:00.514605    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"
Nov 03 03:08:00.676856 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:00.676822    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eded8d0f-e8e1-4410-baa9-e2982338f3e3-kube-api-access-sbm6v\" (UniqueName: \"kubernetes.io/projected/eded8d0f-e8e1-4410-baa9-e2982338f3e3-kube-api-access-sbm6v\") pod \"eded8d0f-e8e1-4410-baa9-e2982338f3e3\" (UID: \"eded8d0f-e8e1-4410-baa9-e2982338f3e3\") "
Nov 03 03:08:00.677276 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:00.677240    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eded8d0f-e8e1-4410-baa9-e2982338f3e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eded8d0f-e8e1-4410-baa9-e2982338f3e3-maas-api-tls\") pod \"eded8d0f-e8e1-4410-baa9-e2982338f3e3\" (UID: \"eded8d0f-e8e1-4410-baa9-e2982338f3e3\") "
Nov 03 03:08:00.677427 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:00.677409    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eded8d0f-e8e1-4410-baa9-e2982338f3e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eded8d0f-e8e1-4410-baa9-e2982338f3e3-metrics-certs\") pod \"eded8d0f-e8e1-4410-baa9-e2982338f3e3\" (UID: \"eded8d0f-e8e1-4410-baa9-e2982338f3e3\") "
Nov 03 03:08:00.677547 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:00.677528    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eded8d0f-e8e1-4410-baa9-e2982338f3e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eded8d0f-e8e1-4410-baa9-e2982338f3e3-openshift-service-ca\") pod \"eded8d0f-e8e1-4410-baa9-e2982338f3e3\" (UID: \"eded8d0f-e8e1-4410-baa9-e2982338f3e3\") "
Nov 03 03:08:00.678004 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:00.677974    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eded8d0f-e8e1-4410-baa9-e2982338f3e3-openshift-service-ca" pod "eded8d0f-e8e1-4410-baa9-e2982338f3e3" (UID: "eded8d0f-e8e1-4410-baa9-e2982338f3e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:08:00.679176 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:00.679148    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eded8d0f-e8e1-4410-baa9-e2982338f3e3-kube-api-access-sbm6v" pod "eded8d0f-e8e1-4410-baa9-e2982338f3e3" (UID: "eded8d0f-e8e1-4410-baa9-e2982338f3e3"). InnerVolumeSpecName "kube-api-access-sbm6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:08:00.679541 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:00.679516    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eded8d0f-e8e1-4410-baa9-e2982338f3e3-metrics-certs" pod "eded8d0f-e8e1-4410-baa9-e2982338f3e3" (UID: "eded8d0f-e8e1-4410-baa9-e2982338f3e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:08:00.682290 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:00.682259    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eded8d0f-e8e1-4410-baa9-e2982338f3e3-maas-api-tls" pod "eded8d0f-e8e1-4410-baa9-e2982338f3e3" (UID: "eded8d0f-e8e1-4410-baa9-e2982338f3e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:08:00.719704 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:08:00.719657    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:08:00.779507 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:00.779470    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eded8d0f-e8e1-4410-baa9-e2982338f3e3-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:00.779507 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:00.779505    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eded8d0f-e8e1-4410-baa9-e2982338f3e3-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:00.779771 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:00.779515    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbm6v\" (UniqueName: \"kubernetes.io/projected/eded8d0f-e8e1-4410-baa9-e2982338f3e3-kube-api-access-sbm6v\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:00.779771 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:00.779525    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eded8d0f-e8e1-4410-baa9-e2982338f3e3-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:01.285889 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:01.285854    5005 generic.go:356] "Generic (PLEG): container finished" podID="eded8d0f-e8e1-4410-baa9-e2982338f3e3" containerID="846eea1f15d4b0fb2c1dfffac87d0646576570dbfe329d5c36642f8de3893189" exitCode=0
Nov 03 03:08:01.286339 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:01.285903    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"
Nov 03 03:08:01.286339 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:01.285927    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw" event={"ID":"eded8d0f-e8e1-4410-baa9-e2982338f3e3","Type":"ContainerDied","Data":"846eea1f15d4b0fb2c1dfffac87d0646576570dbfe329d5c36642f8de3893189"}
Nov 03 03:08:01.286339 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:01.285969    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw" event={"ID":"eded8d0f-e8e1-4410-baa9-e2982338f3e3","Type":"ContainerDied","Data":"9abaebbc8c708a7e4d31fa831411c520447d72b8f08a69746b3d0aef3f1588cc"}
Nov 03 03:08:01.286339 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:01.285989    5005 scope.go:122] "RemoveContainer" containerID="846eea1f15d4b0fb2c1dfffac87d0646576570dbfe329d5c36642f8de3893189"
Nov 03 03:08:01.295929 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:01.295908    5005 scope.go:122] "RemoveContainer" containerID="846eea1f15d4b0fb2c1dfffac87d0646576570dbfe329d5c36642f8de3893189"
Nov 03 03:08:01.296190 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:08:01.296170    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"846eea1f15d4b0fb2c1dfffac87d0646576570dbfe329d5c36642f8de3893189\": container with ID starting with 846eea1f15d4b0fb2c1dfffac87d0646576570dbfe329d5c36642f8de3893189 not found: ID does not exist" containerID="846eea1f15d4b0fb2c1dfffac87d0646576570dbfe329d5c36642f8de3893189"
Nov 03 03:08:01.296260 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:01.296204    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"846eea1f15d4b0fb2c1dfffac87d0646576570dbfe329d5c36642f8de3893189"} err="failed to get container status \"846eea1f15d4b0fb2c1dfffac87d0646576570dbfe329d5c36642f8de3893189\": rpc error: code = NotFound desc = could not find container \"846eea1f15d4b0fb2c1dfffac87d0646576570dbfe329d5c36642f8de3893189\": container with ID starting with 846eea1f15d4b0fb2c1dfffac87d0646576570dbfe329d5c36642f8de3893189 not found: ID does not exist"
Nov 03 03:08:01.443783 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:01.443705    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"]
Nov 03 03:08:01.489895 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:01.489860    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7273af-749dd66579-h6xxw"]
Nov 03 03:08:01.628228 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:01.628190    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"]
Nov 03 03:08:03.110451 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:03.110419    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eded8d0f-e8e1-4410-baa9-e2982338f3e3" path="/var/lib/kubelet/pods/eded8d0f-e8e1-4410-baa9-e2982338f3e3/volumes"
Nov 03 03:08:16.656590 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:16.656535    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td" podUID="7f39ea17-c4d7-4de2-bb69-c7d3012c5f77" containerName="main" containerID="cri-o://12bc1b6324d761be75c7e809d95ad5c8cabc8db4ab1d80fa7aab434181cc09b8" gracePeriod=45
Nov 03 03:08:17.036750 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.036714    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 03:08:17.101334 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.101299    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-home\" (UniqueName: \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-home\") pod \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") "
Nov 03 03:08:17.101524 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.101438    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-kube-api-access-g4vvx\" (UniqueName: \"kubernetes.io/projected/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-kube-api-access-g4vvx\") pod \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") "
Nov 03 03:08:17.101524 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.101519    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-tmp-dir\") pod \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") "
Nov 03 03:08:17.101649 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.101584    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-tls-certs\") pod \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") "
Nov 03 03:08:17.101649 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.101610    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-home" pod "7f39ea17-c4d7-4de2-bb69-c7d3012c5f77" (UID: "7f39ea17-c4d7-4de2-bb69-c7d3012c5f77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:08:17.101649 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.101631    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-dshm\") pod \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") "
Nov 03 03:08:17.101853 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.101739    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-model-cache\") pod \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\" (UID: \"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77\") "
Nov 03 03:08:17.102128 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.102105    5005 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-home\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:17.102361 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.102322    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-tmp-dir" pod "7f39ea17-c4d7-4de2-bb69-c7d3012c5f77" (UID: "7f39ea17-c4d7-4de2-bb69-c7d3012c5f77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:08:17.102361 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.102339    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-model-cache" pod "7f39ea17-c4d7-4de2-bb69-c7d3012c5f77" (UID: "7f39ea17-c4d7-4de2-bb69-c7d3012c5f77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:08:17.104535 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.104506    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-dshm" pod "7f39ea17-c4d7-4de2-bb69-c7d3012c5f77" (UID: "7f39ea17-c4d7-4de2-bb69-c7d3012c5f77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:08:17.104648 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.104549    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-kube-api-access-g4vvx" pod "7f39ea17-c4d7-4de2-bb69-c7d3012c5f77" (UID: "7f39ea17-c4d7-4de2-bb69-c7d3012c5f77"). InnerVolumeSpecName "kube-api-access-g4vvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:08:17.105241 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.105222    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-tls-certs" pod "7f39ea17-c4d7-4de2-bb69-c7d3012c5f77" (UID: "7f39ea17-c4d7-4de2-bb69-c7d3012c5f77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:08:17.204804 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.204300    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4vvx\" (UniqueName: \"kubernetes.io/projected/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-kube-api-access-g4vvx\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:17.204804 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.204356    5005 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-tmp-dir\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:17.204804 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.204382    5005 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-tls-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:17.204804 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.204398    5005 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-dshm\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:17.204804 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.204413    5005 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77-model-cache\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:17.412413 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.412380    5005 generic.go:356] "Generic (PLEG): container finished" podID="7f39ea17-c4d7-4de2-bb69-c7d3012c5f77" containerID="12bc1b6324d761be75c7e809d95ad5c8cabc8db4ab1d80fa7aab434181cc09b8" exitCode=0
Nov 03 03:08:17.412599 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.412423    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"
Nov 03 03:08:17.412599 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.412459    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td" event={"ID":"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77","Type":"ContainerDied","Data":"12bc1b6324d761be75c7e809d95ad5c8cabc8db4ab1d80fa7aab434181cc09b8"}
Nov 03 03:08:17.412599 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.412509    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td" event={"ID":"7f39ea17-c4d7-4de2-bb69-c7d3012c5f77","Type":"ContainerDied","Data":"1282d4636f6214746fb2cdc6bbc0f230147777bc94892eb7a0055a34b1636260"}
Nov 03 03:08:17.412599 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.412531    5005 scope.go:122] "RemoveContainer" containerID="12bc1b6324d761be75c7e809d95ad5c8cabc8db4ab1d80fa7aab434181cc09b8"
Nov 03 03:08:17.422732 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.422685    5005 scope.go:122] "RemoveContainer" containerID="12bc1b6324d761be75c7e809d95ad5c8cabc8db4ab1d80fa7aab434181cc09b8"
Nov 03 03:08:17.423015 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:08:17.422988    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12bc1b6324d761be75c7e809d95ad5c8cabc8db4ab1d80fa7aab434181cc09b8\": container with ID starting with 12bc1b6324d761be75c7e809d95ad5c8cabc8db4ab1d80fa7aab434181cc09b8 not found: ID does not exist" containerID="12bc1b6324d761be75c7e809d95ad5c8cabc8db4ab1d80fa7aab434181cc09b8"
Nov 03 03:08:17.423072 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.423019    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12bc1b6324d761be75c7e809d95ad5c8cabc8db4ab1d80fa7aab434181cc09b8"} err="failed to get container status \"12bc1b6324d761be75c7e809d95ad5c8cabc8db4ab1d80fa7aab434181cc09b8\": rpc error: code = NotFound desc = could not find container \"12bc1b6324d761be75c7e809d95ad5c8cabc8db4ab1d80fa7aab434181cc09b8\": container with ID starting with 12bc1b6324d761be75c7e809d95ad5c8cabc8db4ab1d80fa7aab434181cc09b8 not found: ID does not exist"
Nov 03 03:08:17.566922 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.566882    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"]
Nov 03 03:08:17.569929 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:17.569900    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td"]
Nov 03 03:08:18.018330 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:18.018289    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c7d758/distinct-w4-c7d758-kserve-d4c9f769f-6v4td" podUID="7f39ea17-c4d7-4de2-bb69-c7d3012c5f77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:08:19.109943 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:19.109908    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f39ea17-c4d7-4de2-bb69-c7d3012c5f77" path="/var/lib/kubelet/pods/7f39ea17-c4d7-4de2-bb69-c7d3012c5f77/volumes"
Nov 03 03:08:28.702275 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:28.702176    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"]
Nov 03 03:08:28.702829 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:28.702449    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt" podUID="55f0246d-7f9d-4581-bd2b-c98e20e97676" containerName="istio-proxy" containerID="cri-o://7a5d27b54f44e58debd75bb5b3db172a4c457812c2613d8e54bd0e773c0cacf5" gracePeriod=30
Nov 03 03:08:30.718022 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:08:30.717978    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:08:31.398063 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:31.398024    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-99swt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 03:08:31.398254 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:31.398097    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt" podUID="55f0246d-7f9d-4581-bd2b-c98e20e97676" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 03:08:33.968087 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:33.968060    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 03:08:34.143109 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.143072    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-envoy\") pod \"55f0246d-7f9d-4581-bd2b-c98e20e97676\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") "
Nov 03 03:08:34.143411 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.143385    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-credential-socket\") pod \"55f0246d-7f9d-4581-bd2b-c98e20e97676\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") "
Nov 03 03:08:34.143623 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.143598    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f0246d-7f9d-4581-bd2b-c98e20e97676-kube-api-access-2gc4g\" (UniqueName: \"kubernetes.io/projected/55f0246d-7f9d-4581-bd2b-c98e20e97676-kube-api-access-2gc4g\") pod \"55f0246d-7f9d-4581-bd2b-c98e20e97676\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") "
Nov 03 03:08:34.143759 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.143668    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-data\") pod \"55f0246d-7f9d-4581-bd2b-c98e20e97676\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") "
Nov 03 03:08:34.143759 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.143752    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-workload-certs\") pod \"55f0246d-7f9d-4581-bd2b-c98e20e97676\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") "
Nov 03 03:08:34.143951 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.143867    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-podinfo\") pod \"55f0246d-7f9d-4581-bd2b-c98e20e97676\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") "
Nov 03 03:08:34.143951 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.143925    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-token\" (UniqueName: \"kubernetes.io/projected/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-token\") pod \"55f0246d-7f9d-4581-bd2b-c98e20e97676\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") "
Nov 03 03:08:34.144066 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.144013    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-workload-socket\") pod \"55f0246d-7f9d-4581-bd2b-c98e20e97676\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") "
Nov 03 03:08:34.144066 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.144029    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-workload-certs" pod "55f0246d-7f9d-4581-bd2b-c98e20e97676" (UID: "55f0246d-7f9d-4581-bd2b-c98e20e97676"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:08:34.144170 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.144068    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f0246d-7f9d-4581-bd2b-c98e20e97676-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f0246d-7f9d-4581-bd2b-c98e20e97676-istiod-ca-cert\") pod \"55f0246d-7f9d-4581-bd2b-c98e20e97676\" (UID: \"55f0246d-7f9d-4581-bd2b-c98e20e97676\") "
Nov 03 03:08:34.146894 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.144337    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-credential-socket" pod "55f0246d-7f9d-4581-bd2b-c98e20e97676" (UID: "55f0246d-7f9d-4581-bd2b-c98e20e97676"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:08:34.146894 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.144352    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-workload-socket" pod "55f0246d-7f9d-4581-bd2b-c98e20e97676" (UID: "55f0246d-7f9d-4581-bd2b-c98e20e97676"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:08:34.146894 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.144630    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-data" pod "55f0246d-7f9d-4581-bd2b-c98e20e97676" (UID: "55f0246d-7f9d-4581-bd2b-c98e20e97676"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:08:34.146894 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.144659    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:34.146894 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.144685    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:34.146894 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.144742    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:34.146894 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.144815    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f0246d-7f9d-4581-bd2b-c98e20e97676-istiod-ca-cert" pod "55f0246d-7f9d-4581-bd2b-c98e20e97676" (UID: "55f0246d-7f9d-4581-bd2b-c98e20e97676"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:08:34.146894 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.146780    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-envoy" pod "55f0246d-7f9d-4581-bd2b-c98e20e97676" (UID: "55f0246d-7f9d-4581-bd2b-c98e20e97676"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:08:34.147391 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.147371    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f0246d-7f9d-4581-bd2b-c98e20e97676-kube-api-access-2gc4g" pod "55f0246d-7f9d-4581-bd2b-c98e20e97676" (UID: "55f0246d-7f9d-4581-bd2b-c98e20e97676"). InnerVolumeSpecName "kube-api-access-2gc4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:08:34.147797 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.147779    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-token" pod "55f0246d-7f9d-4581-bd2b-c98e20e97676" (UID: "55f0246d-7f9d-4581-bd2b-c98e20e97676"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:08:34.147874 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.147857    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-podinfo" pod "55f0246d-7f9d-4581-bd2b-c98e20e97676" (UID: "55f0246d-7f9d-4581-bd2b-c98e20e97676"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:08:34.246525 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.246486    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:34.246525 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.246523    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f0246d-7f9d-4581-bd2b-c98e20e97676-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:34.246840 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.246541    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:34.246840 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.246557    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gc4g\" (UniqueName: \"kubernetes.io/projected/55f0246d-7f9d-4581-bd2b-c98e20e97676-kube-api-access-2gc4g\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:34.246840 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.246572    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:34.246840 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.246585    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f0246d-7f9d-4581-bd2b-c98e20e97676-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:34.522221 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.522183    5005 generic.go:356] "Generic (PLEG): container finished" podID="55f0246d-7f9d-4581-bd2b-c98e20e97676" containerID="7a5d27b54f44e58debd75bb5b3db172a4c457812c2613d8e54bd0e773c0cacf5" exitCode=0
Nov 03 03:08:34.522392 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.522254    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"
Nov 03 03:08:34.522392 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.522259    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt" event={"ID":"55f0246d-7f9d-4581-bd2b-c98e20e97676","Type":"ContainerDied","Data":"7a5d27b54f44e58debd75bb5b3db172a4c457812c2613d8e54bd0e773c0cacf5"}
Nov 03 03:08:34.522392 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.522342    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt" event={"ID":"55f0246d-7f9d-4581-bd2b-c98e20e97676","Type":"ContainerDied","Data":"c9c158965c9c96b95021f90dac913332a10ebfe594076ec2fd5451c55bdd2009"}
Nov 03 03:08:34.522392 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.522365    5005 scope.go:122] "RemoveContainer" containerID="7a5d27b54f44e58debd75bb5b3db172a4c457812c2613d8e54bd0e773c0cacf5"
Nov 03 03:08:34.533530 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.533511    5005 scope.go:122] "RemoveContainer" containerID="7a5d27b54f44e58debd75bb5b3db172a4c457812c2613d8e54bd0e773c0cacf5"
Nov 03 03:08:34.533819 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:08:34.533795    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a5d27b54f44e58debd75bb5b3db172a4c457812c2613d8e54bd0e773c0cacf5\": container with ID starting with 7a5d27b54f44e58debd75bb5b3db172a4c457812c2613d8e54bd0e773c0cacf5 not found: ID does not exist" containerID="7a5d27b54f44e58debd75bb5b3db172a4c457812c2613d8e54bd0e773c0cacf5"
Nov 03 03:08:34.533900 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.533830    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a5d27b54f44e58debd75bb5b3db172a4c457812c2613d8e54bd0e773c0cacf5"} err="failed to get container status \"7a5d27b54f44e58debd75bb5b3db172a4c457812c2613d8e54bd0e773c0cacf5\": rpc error: code = NotFound desc = could not find container \"7a5d27b54f44e58debd75bb5b3db172a4c457812c2613d8e54bd0e773c0cacf5\": container with ID starting with 7a5d27b54f44e58debd75bb5b3db172a4c457812c2613d8e54bd0e773c0cacf5 not found: ID does not exist"
Nov 03 03:08:34.678798 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.678749    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"]
Nov 03 03:08:34.725488 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:34.725455    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-99swt"]
Nov 03 03:08:35.109886 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:08:35.109855    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f0246d-7f9d-4581-bd2b-c98e20e97676" path="/var/lib/kubelet/pods/55f0246d-7f9d-4581-bd2b-c98e20e97676/volumes"
Nov 03 03:09:00.715655 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:09:00.715616    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:09:01.374124 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:01.374085    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"]
Nov 03 03:09:01.374397 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:01.374291    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k" podUID="43baeed2-8514-4f19-90c6-e74f971f32b0" containerName="maas-api" containerID="cri-o://037c1cbf55366f1451cb52d57fc3c6838876bd1927247a1e1487d0c9f7ab845e" gracePeriod=30
Nov 03 03:09:01.668620 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:01.668586    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"
Nov 03 03:09:01.710406 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:01.710369    5005 generic.go:356] "Generic (PLEG): container finished" podID="43baeed2-8514-4f19-90c6-e74f971f32b0" containerID="037c1cbf55366f1451cb52d57fc3c6838876bd1927247a1e1487d0c9f7ab845e" exitCode=0
Nov 03 03:09:01.710577 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:01.710416    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"
Nov 03 03:09:01.710577 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:01.710450    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k" event={"ID":"43baeed2-8514-4f19-90c6-e74f971f32b0","Type":"ContainerDied","Data":"037c1cbf55366f1451cb52d57fc3c6838876bd1927247a1e1487d0c9f7ab845e"}
Nov 03 03:09:01.710577 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:01.710508    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k" event={"ID":"43baeed2-8514-4f19-90c6-e74f971f32b0","Type":"ContainerDied","Data":"c57c291f4619bba92109f53ac9f9c570e17c222f46929082ed830c3c602dd620"}
Nov 03 03:09:01.710577 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:01.710534    5005 scope.go:122] "RemoveContainer" containerID="037c1cbf55366f1451cb52d57fc3c6838876bd1927247a1e1487d0c9f7ab845e"
Nov 03 03:09:01.723537 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:01.723510    5005 scope.go:122] "RemoveContainer" containerID="037c1cbf55366f1451cb52d57fc3c6838876bd1927247a1e1487d0c9f7ab845e"
Nov 03 03:09:01.723982 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:09:01.723865    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"037c1cbf55366f1451cb52d57fc3c6838876bd1927247a1e1487d0c9f7ab845e\": container with ID starting with 037c1cbf55366f1451cb52d57fc3c6838876bd1927247a1e1487d0c9f7ab845e not found: ID does not exist" containerID="037c1cbf55366f1451cb52d57fc3c6838876bd1927247a1e1487d0c9f7ab845e"
Nov 03 03:09:01.723982 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:01.723905    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037c1cbf55366f1451cb52d57fc3c6838876bd1927247a1e1487d0c9f7ab845e"} err="failed to get container status \"037c1cbf55366f1451cb52d57fc3c6838876bd1927247a1e1487d0c9f7ab845e\": rpc error: code = NotFound desc = could not find container \"037c1cbf55366f1451cb52d57fc3c6838876bd1927247a1e1487d0c9f7ab845e\": container with ID starting with 037c1cbf55366f1451cb52d57fc3c6838876bd1927247a1e1487d0c9f7ab845e not found: ID does not exist"
Nov 03 03:09:01.727280 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:01.727259    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43baeed2-8514-4f19-90c6-e74f971f32b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43baeed2-8514-4f19-90c6-e74f971f32b0-openshift-service-ca\") pod \"43baeed2-8514-4f19-90c6-e74f971f32b0\" (UID: \"43baeed2-8514-4f19-90c6-e74f971f32b0\") "
Nov 03 03:09:01.727406 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:01.727361    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43baeed2-8514-4f19-90c6-e74f971f32b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/43baeed2-8514-4f19-90c6-e74f971f32b0-metrics-certs\") pod \"43baeed2-8514-4f19-90c6-e74f971f32b0\" (UID: \"43baeed2-8514-4f19-90c6-e74f971f32b0\") "
Nov 03 03:09:01.727477 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:01.727426    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43baeed2-8514-4f19-90c6-e74f971f32b0-kube-api-access-4wkwt\" (UniqueName: \"kubernetes.io/projected/43baeed2-8514-4f19-90c6-e74f971f32b0-kube-api-access-4wkwt\") pod \"43baeed2-8514-4f19-90c6-e74f971f32b0\" (UID: \"43baeed2-8514-4f19-90c6-e74f971f32b0\") "
Nov 03 03:09:01.727596 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:01.727573    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43baeed2-8514-4f19-90c6-e74f971f32b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43baeed2-8514-4f19-90c6-e74f971f32b0-maas-api-tls\") pod \"43baeed2-8514-4f19-90c6-e74f971f32b0\" (UID: \"43baeed2-8514-4f19-90c6-e74f971f32b0\") "
Nov 03 03:09:01.727668 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:01.727600    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43baeed2-8514-4f19-90c6-e74f971f32b0-openshift-service-ca" pod "43baeed2-8514-4f19-90c6-e74f971f32b0" (UID: "43baeed2-8514-4f19-90c6-e74f971f32b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:09:01.728145 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:01.728122    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43baeed2-8514-4f19-90c6-e74f971f32b0-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:09:01.729786 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:01.729756    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43baeed2-8514-4f19-90c6-e74f971f32b0-maas-api-tls" pod "43baeed2-8514-4f19-90c6-e74f971f32b0" (UID: "43baeed2-8514-4f19-90c6-e74f971f32b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:09:01.730095 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:01.730073    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43baeed2-8514-4f19-90c6-e74f971f32b0-kube-api-access-4wkwt" pod "43baeed2-8514-4f19-90c6-e74f971f32b0" (UID: "43baeed2-8514-4f19-90c6-e74f971f32b0"). InnerVolumeSpecName "kube-api-access-4wkwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:09:01.730156 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:01.730095    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43baeed2-8514-4f19-90c6-e74f971f32b0-metrics-certs" pod "43baeed2-8514-4f19-90c6-e74f971f32b0" (UID: "43baeed2-8514-4f19-90c6-e74f971f32b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:09:01.828960 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:01.828926    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43baeed2-8514-4f19-90c6-e74f971f32b0-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:09:01.828960 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:01.828958    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wkwt\" (UniqueName: \"kubernetes.io/projected/43baeed2-8514-4f19-90c6-e74f971f32b0-kube-api-access-4wkwt\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:09:01.829168 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:01.828972    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43baeed2-8514-4f19-90c6-e74f971f32b0-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:09:02.125365 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:02.125321    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"]
Nov 03 03:09:02.129371 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:02.129340    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a0e72e-5d64b4567f-4gt8k"]
Nov 03 03:09:03.109922 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:03.109888    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43baeed2-8514-4f19-90c6-e74f971f32b0" path="/var/lib/kubelet/pods/43baeed2-8514-4f19-90c6-e74f971f32b0/volumes"
Nov 03 03:09:17.235868 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.235830    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"]
Nov 03 03:09:17.236331 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.236035    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w" podUID="fbdeec42-7115-4361-ac6f-1874ffded23c" containerName="maas-api" containerID="cri-o://e4df71d557a63ffc96a498ba7c04c66861643797248eaed313af69878434127a" gracePeriod=30
Nov 03 03:09:17.520239 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.520214    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"
Nov 03 03:09:17.629273 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.629232    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbdeec42-7115-4361-ac6f-1874ffded23c-kube-api-access-l9lqz\" (UniqueName: \"kubernetes.io/projected/fbdeec42-7115-4361-ac6f-1874ffded23c-kube-api-access-l9lqz\") pod \"fbdeec42-7115-4361-ac6f-1874ffded23c\" (UID: \"fbdeec42-7115-4361-ac6f-1874ffded23c\") "
Nov 03 03:09:17.629559 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.629536    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbdeec42-7115-4361-ac6f-1874ffded23c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbdeec42-7115-4361-ac6f-1874ffded23c-maas-api-tls\") pod \"fbdeec42-7115-4361-ac6f-1874ffded23c\" (UID: \"fbdeec42-7115-4361-ac6f-1874ffded23c\") "
Nov 03 03:09:17.636262 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.629806    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbdeec42-7115-4361-ac6f-1874ffded23c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbdeec42-7115-4361-ac6f-1874ffded23c-metrics-certs\") pod \"fbdeec42-7115-4361-ac6f-1874ffded23c\" (UID: \"fbdeec42-7115-4361-ac6f-1874ffded23c\") "
Nov 03 03:09:17.636262 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.629966    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbdeec42-7115-4361-ac6f-1874ffded23c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdeec42-7115-4361-ac6f-1874ffded23c-openshift-service-ca\") pod \"fbdeec42-7115-4361-ac6f-1874ffded23c\" (UID: \"fbdeec42-7115-4361-ac6f-1874ffded23c\") "
Nov 03 03:09:17.636760 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.636724    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbdeec42-7115-4361-ac6f-1874ffded23c-openshift-service-ca" pod "fbdeec42-7115-4361-ac6f-1874ffded23c" (UID: "fbdeec42-7115-4361-ac6f-1874ffded23c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:09:17.643925 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.643889    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbdeec42-7115-4361-ac6f-1874ffded23c-metrics-certs" pod "fbdeec42-7115-4361-ac6f-1874ffded23c" (UID: "fbdeec42-7115-4361-ac6f-1874ffded23c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:09:17.644399 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.644373    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbdeec42-7115-4361-ac6f-1874ffded23c-maas-api-tls" pod "fbdeec42-7115-4361-ac6f-1874ffded23c" (UID: "fbdeec42-7115-4361-ac6f-1874ffded23c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:09:17.645589 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.645553    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdeec42-7115-4361-ac6f-1874ffded23c-kube-api-access-l9lqz" pod "fbdeec42-7115-4361-ac6f-1874ffded23c" (UID: "fbdeec42-7115-4361-ac6f-1874ffded23c"). InnerVolumeSpecName "kube-api-access-l9lqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:09:17.732297 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.732256    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9lqz\" (UniqueName: \"kubernetes.io/projected/fbdeec42-7115-4361-ac6f-1874ffded23c-kube-api-access-l9lqz\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:09:17.732297 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.732295    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbdeec42-7115-4361-ac6f-1874ffded23c-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:09:17.732499 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.732310    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbdeec42-7115-4361-ac6f-1874ffded23c-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:09:17.732499 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.732325    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdeec42-7115-4361-ac6f-1874ffded23c-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:09:17.829216 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.829171    5005 generic.go:356] "Generic (PLEG): container finished" podID="fbdeec42-7115-4361-ac6f-1874ffded23c" containerID="e4df71d557a63ffc96a498ba7c04c66861643797248eaed313af69878434127a" exitCode=0
Nov 03 03:09:17.829424 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.829309    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w" event={"ID":"fbdeec42-7115-4361-ac6f-1874ffded23c","Type":"ContainerDied","Data":"e4df71d557a63ffc96a498ba7c04c66861643797248eaed313af69878434127a"}
Nov 03 03:09:17.829424 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.829388    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w" event={"ID":"fbdeec42-7115-4361-ac6f-1874ffded23c","Type":"ContainerDied","Data":"72835cd833dc59a9475efd57993395471ba0eb4175abfa5c0eb5674be3a37d1e"}
Nov 03 03:09:17.829531 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.829425    5005 scope.go:122] "RemoveContainer" containerID="e4df71d557a63ffc96a498ba7c04c66861643797248eaed313af69878434127a"
Nov 03 03:09:17.829631 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.829613    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"
Nov 03 03:09:17.840916 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.840892    5005 scope.go:122] "RemoveContainer" containerID="e4df71d557a63ffc96a498ba7c04c66861643797248eaed313af69878434127a"
Nov 03 03:09:17.841239 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:09:17.841215    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4df71d557a63ffc96a498ba7c04c66861643797248eaed313af69878434127a\": container with ID starting with e4df71d557a63ffc96a498ba7c04c66861643797248eaed313af69878434127a not found: ID does not exist" containerID="e4df71d557a63ffc96a498ba7c04c66861643797248eaed313af69878434127a"
Nov 03 03:09:17.841278 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.841250    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4df71d557a63ffc96a498ba7c04c66861643797248eaed313af69878434127a"} err="failed to get container status \"e4df71d557a63ffc96a498ba7c04c66861643797248eaed313af69878434127a\": rpc error: code = NotFound desc = could not find container \"e4df71d557a63ffc96a498ba7c04c66861643797248eaed313af69878434127a\": container with ID starting with e4df71d557a63ffc96a498ba7c04c66861643797248eaed313af69878434127a not found: ID does not exist"
Nov 03 03:09:17.984279 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.984222    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"]
Nov 03 03:09:17.987647 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:17.987602    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7d758-566db48d7d-p8b8w"]
Nov 03 03:09:19.109913 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:09:19.109876    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbdeec42-7115-4361-ac6f-1874ffded23c" path="/var/lib/kubelet/pods/fbdeec42-7115-4361-ac6f-1874ffded23c/volumes"
Nov 03 03:09:30.715286 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:09:30.715240    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:10:00.714506 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:10:00.714417    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:10:30.714592 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:10:30.714553    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:11:00.715062 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:11:00.715016    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:11:13.808555 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:13.808450    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 03:11:13.837760 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:13.837732    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 03:11:30.718301 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:11:30.718207    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:11:37.423372 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:37.423336    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"]
Nov 03 03:11:37.423813 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:37.423553    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn" podUID="c58a3573-fd82-4f92-829b-475591248445" containerName="maas-api" containerID="cri-o://2afa0286e14606cbeb8df6667f5fbd80aad43e933bd37d7cfacf1ff3aac60923" gracePeriod=30
Nov 03 03:11:37.715785 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:37.715757    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"
Nov 03 03:11:37.741321 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:37.741288    5005 generic.go:356] "Generic (PLEG): container finished" podID="c58a3573-fd82-4f92-829b-475591248445" containerID="2afa0286e14606cbeb8df6667f5fbd80aad43e933bd37d7cfacf1ff3aac60923" exitCode=0
Nov 03 03:11:37.741493 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:37.741339    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"
Nov 03 03:11:37.741493 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:37.741342    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn" event={"ID":"c58a3573-fd82-4f92-829b-475591248445","Type":"ContainerDied","Data":"2afa0286e14606cbeb8df6667f5fbd80aad43e933bd37d7cfacf1ff3aac60923"}
Nov 03 03:11:37.741493 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:37.741472    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn" event={"ID":"c58a3573-fd82-4f92-829b-475591248445","Type":"ContainerDied","Data":"45d3f5d723654022a9fc8c7ef7f4b53bf6b0d027c001de5d05edbfefbb0fa19a"}
Nov 03 03:11:37.741621 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:37.741500    5005 scope.go:122] "RemoveContainer" containerID="2afa0286e14606cbeb8df6667f5fbd80aad43e933bd37d7cfacf1ff3aac60923"
Nov 03 03:11:37.751416 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:37.751394    5005 scope.go:122] "RemoveContainer" containerID="2afa0286e14606cbeb8df6667f5fbd80aad43e933bd37d7cfacf1ff3aac60923"
Nov 03 03:11:37.751741 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:11:37.751685    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2afa0286e14606cbeb8df6667f5fbd80aad43e933bd37d7cfacf1ff3aac60923\": container with ID starting with 2afa0286e14606cbeb8df6667f5fbd80aad43e933bd37d7cfacf1ff3aac60923 not found: ID does not exist" containerID="2afa0286e14606cbeb8df6667f5fbd80aad43e933bd37d7cfacf1ff3aac60923"
Nov 03 03:11:37.751842 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:37.751755    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2afa0286e14606cbeb8df6667f5fbd80aad43e933bd37d7cfacf1ff3aac60923"} err="failed to get container status \"2afa0286e14606cbeb8df6667f5fbd80aad43e933bd37d7cfacf1ff3aac60923\": rpc error: code = NotFound desc = could not find container \"2afa0286e14606cbeb8df6667f5fbd80aad43e933bd37d7cfacf1ff3aac60923\": container with ID starting with 2afa0286e14606cbeb8df6667f5fbd80aad43e933bd37d7cfacf1ff3aac60923 not found: ID does not exist"
Nov 03 03:11:37.778796 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:37.778756    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58a3573-fd82-4f92-829b-475591248445-kube-api-access-42pmx\" (UniqueName: \"kubernetes.io/projected/c58a3573-fd82-4f92-829b-475591248445-kube-api-access-42pmx\") pod \"c58a3573-fd82-4f92-829b-475591248445\" (UID: \"c58a3573-fd82-4f92-829b-475591248445\") "
Nov 03 03:11:37.778987 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:37.778851    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c58a3573-fd82-4f92-829b-475591248445-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58a3573-fd82-4f92-829b-475591248445-maas-api-tls\") pod \"c58a3573-fd82-4f92-829b-475591248445\" (UID: \"c58a3573-fd82-4f92-829b-475591248445\") "
Nov 03 03:11:37.778987 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:37.778957    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c58a3573-fd82-4f92-829b-475591248445-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58a3573-fd82-4f92-829b-475591248445-metrics-certs\") pod \"c58a3573-fd82-4f92-829b-475591248445\" (UID: \"c58a3573-fd82-4f92-829b-475591248445\") "
Nov 03 03:11:37.779105 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:37.779038    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c58a3573-fd82-4f92-829b-475591248445-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58a3573-fd82-4f92-829b-475591248445-openshift-service-ca\") pod \"c58a3573-fd82-4f92-829b-475591248445\" (UID: \"c58a3573-fd82-4f92-829b-475591248445\") "
Nov 03 03:11:37.779410 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:37.779373    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c58a3573-fd82-4f92-829b-475591248445-openshift-service-ca" pod "c58a3573-fd82-4f92-829b-475591248445" (UID: "c58a3573-fd82-4f92-829b-475591248445"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:11:37.779546 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:37.779516    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58a3573-fd82-4f92-829b-475591248445-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:11:37.781076 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:37.781042    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58a3573-fd82-4f92-829b-475591248445-kube-api-access-42pmx" pod "c58a3573-fd82-4f92-829b-475591248445" (UID: "c58a3573-fd82-4f92-829b-475591248445"). InnerVolumeSpecName "kube-api-access-42pmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:11:37.781198 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:37.781137    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c58a3573-fd82-4f92-829b-475591248445-maas-api-tls" pod "c58a3573-fd82-4f92-829b-475591248445" (UID: "c58a3573-fd82-4f92-829b-475591248445"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:11:37.781198 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:37.781152    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c58a3573-fd82-4f92-829b-475591248445-metrics-certs" pod "c58a3573-fd82-4f92-829b-475591248445" (UID: "c58a3573-fd82-4f92-829b-475591248445"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:11:37.880340 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:37.880301    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58a3573-fd82-4f92-829b-475591248445-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:11:37.880340 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:37.880338    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58a3573-fd82-4f92-829b-475591248445-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:11:37.880556 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:37.880353    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42pmx\" (UniqueName: \"kubernetes.io/projected/c58a3573-fd82-4f92-829b-475591248445-kube-api-access-42pmx\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:11:38.155859 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:38.155818    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"]
Nov 03 03:11:38.159871 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:38.159842    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-9aebe8-66cff6885c-jz4tn"]
Nov 03 03:11:39.110358 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:39.110316    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c58a3573-fd82-4f92-829b-475591248445" path="/var/lib/kubelet/pods/c58a3573-fd82-4f92-829b-475591248445/volumes"
Nov 03 03:11:53.417848 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:53.417807    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896"]
Nov 03 03:11:53.524783 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:53.524746    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb"]
Nov 03 03:11:58.419176 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:58.419129    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896" podUID="f8f6fafc-574b-47e5-b63a-0eed748772ee" containerName="payload-processing" containerID="cri-o://a3b388ee209d27862e8dfd4a407ef3d71a8feefbd07f3702f4e5ecabb4074809" gracePeriod=25
Nov 03 03:11:58.526044 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:11:58.525972    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb" podUID="394a32fc-4122-44a7-aef5-937f6f691da2" containerName="payload-pre-processing" containerID="cri-o://5ed6c515a6ad98e1bac8e87462dbdfdb4f2a79ab7a0400a06a58d374973dda66" gracePeriod=25
Nov 03 03:12:00.715308 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:12:00.715270    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:12:03.666337 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:03.666293    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896" podUID="f8f6fafc-574b-47e5-b63a-0eed748772ee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 03:12:03.666860 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:03.666339    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb" podUID="394a32fc-4122-44a7-aef5-937f6f691da2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 03:12:08.529045 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:08.529007    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"]
Nov 03 03:12:08.529685 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:08.529278    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv" podUID="1c21f738-1458-4b56-880f-e4ff7fc8123a" containerName="maas-api" containerID="cri-o://59cc5e573ee9c061111eba99a62a1f81e39c273e7c8eb4b5a4632a1909d4cde1" gracePeriod=30
Nov 03 03:12:08.821508 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:08.821479    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"
Nov 03 03:12:08.951591 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:08.951549    5005 generic.go:356] "Generic (PLEG): container finished" podID="1c21f738-1458-4b56-880f-e4ff7fc8123a" containerID="59cc5e573ee9c061111eba99a62a1f81e39c273e7c8eb4b5a4632a1909d4cde1" exitCode=0
Nov 03 03:12:08.951800 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:08.951612    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"
Nov 03 03:12:08.951800 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:08.951625    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv" event={"ID":"1c21f738-1458-4b56-880f-e4ff7fc8123a","Type":"ContainerDied","Data":"59cc5e573ee9c061111eba99a62a1f81e39c273e7c8eb4b5a4632a1909d4cde1"}
Nov 03 03:12:08.951800 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:08.951685    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv" event={"ID":"1c21f738-1458-4b56-880f-e4ff7fc8123a","Type":"ContainerDied","Data":"b0d58d70c3cb504c02c9f5f2da4e3546cf47aff25575bd9c802429a1b56af265"}
Nov 03 03:12:08.951800 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:08.951738    5005 scope.go:122] "RemoveContainer" containerID="59cc5e573ee9c061111eba99a62a1f81e39c273e7c8eb4b5a4632a1909d4cde1"
Nov 03 03:12:08.957391 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:08.957364    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c21f738-1458-4b56-880f-e4ff7fc8123a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c21f738-1458-4b56-880f-e4ff7fc8123a-metrics-certs\") pod \"1c21f738-1458-4b56-880f-e4ff7fc8123a\" (UID: \"1c21f738-1458-4b56-880f-e4ff7fc8123a\") "
Nov 03 03:12:08.957593 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:08.957570    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c21f738-1458-4b56-880f-e4ff7fc8123a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c21f738-1458-4b56-880f-e4ff7fc8123a-openshift-service-ca\") pod \"1c21f738-1458-4b56-880f-e4ff7fc8123a\" (UID: \"1c21f738-1458-4b56-880f-e4ff7fc8123a\") "
Nov 03 03:12:08.957761 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:08.957743    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c21f738-1458-4b56-880f-e4ff7fc8123a-kube-api-access-cxpgb\" (UniqueName: \"kubernetes.io/projected/1c21f738-1458-4b56-880f-e4ff7fc8123a-kube-api-access-cxpgb\") pod \"1c21f738-1458-4b56-880f-e4ff7fc8123a\" (UID: \"1c21f738-1458-4b56-880f-e4ff7fc8123a\") "
Nov 03 03:12:08.957847 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:08.957795    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c21f738-1458-4b56-880f-e4ff7fc8123a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c21f738-1458-4b56-880f-e4ff7fc8123a-maas-api-tls\") pod \"1c21f738-1458-4b56-880f-e4ff7fc8123a\" (UID: \"1c21f738-1458-4b56-880f-e4ff7fc8123a\") "
Nov 03 03:12:08.958210 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:08.957959    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c21f738-1458-4b56-880f-e4ff7fc8123a-openshift-service-ca" pod "1c21f738-1458-4b56-880f-e4ff7fc8123a" (UID: "1c21f738-1458-4b56-880f-e4ff7fc8123a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:12:08.958387 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:08.958363    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c21f738-1458-4b56-880f-e4ff7fc8123a-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:12:08.960047 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:08.960023    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c21f738-1458-4b56-880f-e4ff7fc8123a-metrics-certs" pod "1c21f738-1458-4b56-880f-e4ff7fc8123a" (UID: "1c21f738-1458-4b56-880f-e4ff7fc8123a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:12:08.960363 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:08.960341    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c21f738-1458-4b56-880f-e4ff7fc8123a-kube-api-access-cxpgb" pod "1c21f738-1458-4b56-880f-e4ff7fc8123a" (UID: "1c21f738-1458-4b56-880f-e4ff7fc8123a"). InnerVolumeSpecName "kube-api-access-cxpgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:12:08.964559 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:08.964439    5005 scope.go:122] "RemoveContainer" containerID="59cc5e573ee9c061111eba99a62a1f81e39c273e7c8eb4b5a4632a1909d4cde1"
Nov 03 03:12:08.964870 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:12:08.964849    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59cc5e573ee9c061111eba99a62a1f81e39c273e7c8eb4b5a4632a1909d4cde1\": container with ID starting with 59cc5e573ee9c061111eba99a62a1f81e39c273e7c8eb4b5a4632a1909d4cde1 not found: ID does not exist" containerID="59cc5e573ee9c061111eba99a62a1f81e39c273e7c8eb4b5a4632a1909d4cde1"
Nov 03 03:12:08.964967 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:08.964879    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59cc5e573ee9c061111eba99a62a1f81e39c273e7c8eb4b5a4632a1909d4cde1"} err="failed to get container status \"59cc5e573ee9c061111eba99a62a1f81e39c273e7c8eb4b5a4632a1909d4cde1\": rpc error: code = NotFound desc = could not find container \"59cc5e573ee9c061111eba99a62a1f81e39c273e7c8eb4b5a4632a1909d4cde1\": container with ID starting with 59cc5e573ee9c061111eba99a62a1f81e39c273e7c8eb4b5a4632a1909d4cde1 not found: ID does not exist"
Nov 03 03:12:08.965275 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:08.965247    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c21f738-1458-4b56-880f-e4ff7fc8123a-maas-api-tls" pod "1c21f738-1458-4b56-880f-e4ff7fc8123a" (UID: "1c21f738-1458-4b56-880f-e4ff7fc8123a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:12:09.060242 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:09.060033    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxpgb\" (UniqueName: \"kubernetes.io/projected/1c21f738-1458-4b56-880f-e4ff7fc8123a-kube-api-access-cxpgb\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:12:09.060242 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:09.060074    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c21f738-1458-4b56-880f-e4ff7fc8123a-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:12:09.060242 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:09.060160    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c21f738-1458-4b56-880f-e4ff7fc8123a-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:12:09.360507 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:09.360428    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"]
Nov 03 03:12:09.366783 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:09.366746    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-3a4683-6c7fc8d7dc-8zfcv"]
Nov 03 03:12:11.109814 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:11.109781    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c21f738-1458-4b56-880f-e4ff7fc8123a" path="/var/lib/kubelet/pods/1c21f738-1458-4b56-880f-e4ff7fc8123a/volumes"
Nov 03 03:12:13.666193 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:13.666155    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb" podUID="394a32fc-4122-44a7-aef5-937f6f691da2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 03:12:13.666589 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:13.666155    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896" podUID="f8f6fafc-574b-47e5-b63a-0eed748772ee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 03:12:18.689843 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:18.689775    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896"
Nov 03 03:12:18.783136 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:18.783108    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8f6fafc-574b-47e5-b63a-0eed748772ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f6fafc-574b-47e5-b63a-0eed748772ee-plugins-config-volume\") pod \"f8f6fafc-574b-47e5-b63a-0eed748772ee\" (UID: \"f8f6fafc-574b-47e5-b63a-0eed748772ee\") "
Nov 03 03:12:18.783278 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:18.783206    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f6fafc-574b-47e5-b63a-0eed748772ee-kube-api-access-h6vcm\" (UniqueName: \"kubernetes.io/projected/f8f6fafc-574b-47e5-b63a-0eed748772ee-kube-api-access-h6vcm\") pod \"f8f6fafc-574b-47e5-b63a-0eed748772ee\" (UID: \"f8f6fafc-574b-47e5-b63a-0eed748772ee\") "
Nov 03 03:12:18.783547 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:18.783526    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f6fafc-574b-47e5-b63a-0eed748772ee-plugins-config-volume" pod "f8f6fafc-574b-47e5-b63a-0eed748772ee" (UID: "f8f6fafc-574b-47e5-b63a-0eed748772ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:12:18.783647 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:18.783633    5005 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f6fafc-574b-47e5-b63a-0eed748772ee-plugins-config-volume\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:12:18.785463 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:18.785437    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f6fafc-574b-47e5-b63a-0eed748772ee-kube-api-access-h6vcm" pod "f8f6fafc-574b-47e5-b63a-0eed748772ee" (UID: "f8f6fafc-574b-47e5-b63a-0eed748772ee"). InnerVolumeSpecName "kube-api-access-h6vcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:12:18.791201 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:18.791181    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb"
Nov 03 03:12:18.884260 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:18.884225    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/394a32fc-4122-44a7-aef5-937f6f691da2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/394a32fc-4122-44a7-aef5-937f6f691da2-plugins-config-volume\") pod \"394a32fc-4122-44a7-aef5-937f6f691da2\" (UID: \"394a32fc-4122-44a7-aef5-937f6f691da2\") "
Nov 03 03:12:18.884451 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:18.884333    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394a32fc-4122-44a7-aef5-937f6f691da2-kube-api-access-9p82v\" (UniqueName: \"kubernetes.io/projected/394a32fc-4122-44a7-aef5-937f6f691da2-kube-api-access-9p82v\") pod \"394a32fc-4122-44a7-aef5-937f6f691da2\" (UID: \"394a32fc-4122-44a7-aef5-937f6f691da2\") "
Nov 03 03:12:18.884623 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:18.884601    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394a32fc-4122-44a7-aef5-937f6f691da2-plugins-config-volume" pod "394a32fc-4122-44a7-aef5-937f6f691da2" (UID: "394a32fc-4122-44a7-aef5-937f6f691da2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:12:18.884671 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:18.884650    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6vcm\" (UniqueName: \"kubernetes.io/projected/f8f6fafc-574b-47e5-b63a-0eed748772ee-kube-api-access-h6vcm\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:12:18.886439 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:18.886419    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394a32fc-4122-44a7-aef5-937f6f691da2-kube-api-access-9p82v" pod "394a32fc-4122-44a7-aef5-937f6f691da2" (UID: "394a32fc-4122-44a7-aef5-937f6f691da2"). InnerVolumeSpecName "kube-api-access-9p82v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:12:18.986171 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:18.986078    5005 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/394a32fc-4122-44a7-aef5-937f6f691da2-plugins-config-volume\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:12:18.986171 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:18.986113    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p82v\" (UniqueName: \"kubernetes.io/projected/394a32fc-4122-44a7-aef5-937f6f691da2-kube-api-access-9p82v\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:12:19.010794 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:19.010758    5005 generic.go:356] "Generic (PLEG): container finished" podID="f8f6fafc-574b-47e5-b63a-0eed748772ee" containerID="a3b388ee209d27862e8dfd4a407ef3d71a8feefbd07f3702f4e5ecabb4074809" exitCode=0
Nov 03 03:12:19.010994 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:19.010813    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896"
Nov 03 03:12:19.010994 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:19.010844    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896" event={"ID":"f8f6fafc-574b-47e5-b63a-0eed748772ee","Type":"ContainerDied","Data":"a3b388ee209d27862e8dfd4a407ef3d71a8feefbd07f3702f4e5ecabb4074809"}
Nov 03 03:12:19.010994 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:19.010897    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896" event={"ID":"f8f6fafc-574b-47e5-b63a-0eed748772ee","Type":"ContainerDied","Data":"fbab3d762c1e07c6151fea3847add5724f327e14fa92a976aba8482691601816"}
Nov 03 03:12:19.010994 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:19.010927    5005 scope.go:122] "RemoveContainer" containerID="a3b388ee209d27862e8dfd4a407ef3d71a8feefbd07f3702f4e5ecabb4074809"
Nov 03 03:12:19.012452 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:19.012430    5005 generic.go:356] "Generic (PLEG): container finished" podID="394a32fc-4122-44a7-aef5-937f6f691da2" containerID="5ed6c515a6ad98e1bac8e87462dbdfdb4f2a79ab7a0400a06a58d374973dda66" exitCode=0
Nov 03 03:12:19.012538 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:19.012467    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb"
Nov 03 03:12:19.012538 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:19.012494    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb" event={"ID":"394a32fc-4122-44a7-aef5-937f6f691da2","Type":"ContainerDied","Data":"5ed6c515a6ad98e1bac8e87462dbdfdb4f2a79ab7a0400a06a58d374973dda66"}
Nov 03 03:12:19.012630 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:19.012540    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb" event={"ID":"394a32fc-4122-44a7-aef5-937f6f691da2","Type":"ContainerDied","Data":"f7aa343ce5c70e4dfc800eab8461baf5257efcc83c371b3ee010488a0e3da248"}
Nov 03 03:12:19.022039 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:19.021941    5005 scope.go:122] "RemoveContainer" containerID="a3b388ee209d27862e8dfd4a407ef3d71a8feefbd07f3702f4e5ecabb4074809"
Nov 03 03:12:19.022264 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:12:19.022237    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3b388ee209d27862e8dfd4a407ef3d71a8feefbd07f3702f4e5ecabb4074809\": container with ID starting with a3b388ee209d27862e8dfd4a407ef3d71a8feefbd07f3702f4e5ecabb4074809 not found: ID does not exist" containerID="a3b388ee209d27862e8dfd4a407ef3d71a8feefbd07f3702f4e5ecabb4074809"
Nov 03 03:12:19.022342 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:19.022270    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3b388ee209d27862e8dfd4a407ef3d71a8feefbd07f3702f4e5ecabb4074809"} err="failed to get container status \"a3b388ee209d27862e8dfd4a407ef3d71a8feefbd07f3702f4e5ecabb4074809\": rpc error: code = NotFound desc = could not find container \"a3b388ee209d27862e8dfd4a407ef3d71a8feefbd07f3702f4e5ecabb4074809\": container with ID starting with a3b388ee209d27862e8dfd4a407ef3d71a8feefbd07f3702f4e5ecabb4074809 not found: ID does not exist"
Nov 03 03:12:19.022342 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:19.022296    5005 scope.go:122] "RemoveContainer" containerID="5ed6c515a6ad98e1bac8e87462dbdfdb4f2a79ab7a0400a06a58d374973dda66"
Nov 03 03:12:19.032415 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:19.032391    5005 scope.go:122] "RemoveContainer" containerID="5ed6c515a6ad98e1bac8e87462dbdfdb4f2a79ab7a0400a06a58d374973dda66"
Nov 03 03:12:19.032680 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:12:19.032659    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ed6c515a6ad98e1bac8e87462dbdfdb4f2a79ab7a0400a06a58d374973dda66\": container with ID starting with 5ed6c515a6ad98e1bac8e87462dbdfdb4f2a79ab7a0400a06a58d374973dda66 not found: ID does not exist" containerID="5ed6c515a6ad98e1bac8e87462dbdfdb4f2a79ab7a0400a06a58d374973dda66"
Nov 03 03:12:19.032762 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:19.032702    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ed6c515a6ad98e1bac8e87462dbdfdb4f2a79ab7a0400a06a58d374973dda66"} err="failed to get container status \"5ed6c515a6ad98e1bac8e87462dbdfdb4f2a79ab7a0400a06a58d374973dda66\": rpc error: code = NotFound desc = could not find container \"5ed6c515a6ad98e1bac8e87462dbdfdb4f2a79ab7a0400a06a58d374973dda66\": container with ID starting with 5ed6c515a6ad98e1bac8e87462dbdfdb4f2a79ab7a0400a06a58d374973dda66 not found: ID does not exist"
Nov 03 03:12:19.165204 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:19.165164    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896"]
Nov 03 03:12:19.168336 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:19.168307    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-ffba43-86c9f4ff9-zr896"]
Nov 03 03:12:19.270970 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:19.270934    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb"]
Nov 03 03:12:19.275789 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:19.275762    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-ffba43-855c8db96c-5tqtb"]
Nov 03 03:12:21.112594 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:21.111119    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="394a32fc-4122-44a7-aef5-937f6f691da2" path="/var/lib/kubelet/pods/394a32fc-4122-44a7-aef5-937f6f691da2/volumes"
Nov 03 03:12:21.112594 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:21.111837    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8f6fafc-574b-47e5-b63a-0eed748772ee" path="/var/lib/kubelet/pods/f8f6fafc-574b-47e5-b63a-0eed748772ee/volumes"
Nov 03 03:12:24.621907 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:24.621872    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"]
Nov 03 03:12:24.622331 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:24.622087    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg" podUID="008a4c72-1f2f-4fed-ae45-0ab403315d94" containerName="maas-api" containerID="cri-o://eb1e468ebf8d2bb7871c23c53759eac27e5564d841be196150b72a93458faa58" gracePeriod=30
Nov 03 03:12:24.925041 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:24.925016    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"
Nov 03 03:12:25.052883 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:25.052843    5005 generic.go:356] "Generic (PLEG): container finished" podID="008a4c72-1f2f-4fed-ae45-0ab403315d94" containerID="eb1e468ebf8d2bb7871c23c53759eac27e5564d841be196150b72a93458faa58" exitCode=0
Nov 03 03:12:25.053063 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:25.052892    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"
Nov 03 03:12:25.053063 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:25.052930    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg" event={"ID":"008a4c72-1f2f-4fed-ae45-0ab403315d94","Type":"ContainerDied","Data":"eb1e468ebf8d2bb7871c23c53759eac27e5564d841be196150b72a93458faa58"}
Nov 03 03:12:25.053063 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:25.052999    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg" event={"ID":"008a4c72-1f2f-4fed-ae45-0ab403315d94","Type":"ContainerDied","Data":"1c918cbea6dc6f81d576f8fe2c355ebe898a57e8385f295b114061e42aaa4fcd"}
Nov 03 03:12:25.053063 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:25.053035    5005 scope.go:122] "RemoveContainer" containerID="eb1e468ebf8d2bb7871c23c53759eac27e5564d841be196150b72a93458faa58"
Nov 03 03:12:25.063124 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:25.063106    5005 scope.go:122] "RemoveContainer" containerID="eb1e468ebf8d2bb7871c23c53759eac27e5564d841be196150b72a93458faa58"
Nov 03 03:12:25.063467 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:12:25.063448    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb1e468ebf8d2bb7871c23c53759eac27e5564d841be196150b72a93458faa58\": container with ID starting with eb1e468ebf8d2bb7871c23c53759eac27e5564d841be196150b72a93458faa58 not found: ID does not exist" containerID="eb1e468ebf8d2bb7871c23c53759eac27e5564d841be196150b72a93458faa58"
Nov 03 03:12:25.063535 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:25.063476    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1e468ebf8d2bb7871c23c53759eac27e5564d841be196150b72a93458faa58"} err="failed to get container status \"eb1e468ebf8d2bb7871c23c53759eac27e5564d841be196150b72a93458faa58\": rpc error: code = NotFound desc = could not find container \"eb1e468ebf8d2bb7871c23c53759eac27e5564d841be196150b72a93458faa58\": container with ID starting with eb1e468ebf8d2bb7871c23c53759eac27e5564d841be196150b72a93458faa58 not found: ID does not exist"
Nov 03 03:12:25.076836 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:25.076806    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/008a4c72-1f2f-4fed-ae45-0ab403315d94-metrics-certs\" (UniqueName: \"kubernetes.io/secret/008a4c72-1f2f-4fed-ae45-0ab403315d94-metrics-certs\") pod \"008a4c72-1f2f-4fed-ae45-0ab403315d94\" (UID: \"008a4c72-1f2f-4fed-ae45-0ab403315d94\") "
Nov 03 03:12:25.077007 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:25.076914    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/008a4c72-1f2f-4fed-ae45-0ab403315d94-kube-api-access-qslrh\" (UniqueName: \"kubernetes.io/projected/008a4c72-1f2f-4fed-ae45-0ab403315d94-kube-api-access-qslrh\") pod \"008a4c72-1f2f-4fed-ae45-0ab403315d94\" (UID: \"008a4c72-1f2f-4fed-ae45-0ab403315d94\") "
Nov 03 03:12:25.077007 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:25.076951    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/008a4c72-1f2f-4fed-ae45-0ab403315d94-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/008a4c72-1f2f-4fed-ae45-0ab403315d94-openshift-service-ca\") pod \"008a4c72-1f2f-4fed-ae45-0ab403315d94\" (UID: \"008a4c72-1f2f-4fed-ae45-0ab403315d94\") "
Nov 03 03:12:25.077134 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:25.077041    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/008a4c72-1f2f-4fed-ae45-0ab403315d94-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/008a4c72-1f2f-4fed-ae45-0ab403315d94-maas-api-tls\") pod \"008a4c72-1f2f-4fed-ae45-0ab403315d94\" (UID: \"008a4c72-1f2f-4fed-ae45-0ab403315d94\") "
Nov 03 03:12:25.077412 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:25.077385    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/008a4c72-1f2f-4fed-ae45-0ab403315d94-openshift-service-ca" pod "008a4c72-1f2f-4fed-ae45-0ab403315d94" (UID: "008a4c72-1f2f-4fed-ae45-0ab403315d94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:12:25.077563 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:25.077537    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/008a4c72-1f2f-4fed-ae45-0ab403315d94-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:12:25.079202 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:25.079174    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/008a4c72-1f2f-4fed-ae45-0ab403315d94-metrics-certs" pod "008a4c72-1f2f-4fed-ae45-0ab403315d94" (UID: "008a4c72-1f2f-4fed-ae45-0ab403315d94"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:12:25.079349 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:25.079323    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/008a4c72-1f2f-4fed-ae45-0ab403315d94-kube-api-access-qslrh" pod "008a4c72-1f2f-4fed-ae45-0ab403315d94" (UID: "008a4c72-1f2f-4fed-ae45-0ab403315d94"). InnerVolumeSpecName "kube-api-access-qslrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:12:25.079555 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:25.079539    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/008a4c72-1f2f-4fed-ae45-0ab403315d94-maas-api-tls" pod "008a4c72-1f2f-4fed-ae45-0ab403315d94" (UID: "008a4c72-1f2f-4fed-ae45-0ab403315d94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:12:25.178234 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:25.178158    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/008a4c72-1f2f-4fed-ae45-0ab403315d94-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:12:25.178234 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:25.178190    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qslrh\" (UniqueName: \"kubernetes.io/projected/008a4c72-1f2f-4fed-ae45-0ab403315d94-kube-api-access-qslrh\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:12:25.178234 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:25.178202    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/008a4c72-1f2f-4fed-ae45-0ab403315d94-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:12:25.463411 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:25.463312    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"]
Nov 03 03:12:25.509625 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:25.509585    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ffba43-6fd8f85d84-27dvg"]
Nov 03 03:12:27.110114 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:27.110080    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="008a4c72-1f2f-4fed-ae45-0ab403315d94" path="/var/lib/kubelet/pods/008a4c72-1f2f-4fed-ae45-0ab403315d94/volumes"
Nov 03 03:12:30.713597 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:12:30.713561    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:12:50.391464 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:50.391429    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"]
Nov 03 03:12:50.392016 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:50.391644    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x" podUID="a8b19c9c-11e2-4427-8229-86457df5ebfe" containerName="maas-api" containerID="cri-o://6b040fae38d603bcbb5af0a634459b8d45f0e55e7988acf55c3ae3ccb828250b" gracePeriod=30
Nov 03 03:12:50.678458 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:50.678432    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"
Nov 03 03:12:50.785268 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:50.783182    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b19c9c-11e2-4427-8229-86457df5ebfe-kube-api-access-vkgls\" (UniqueName: \"kubernetes.io/projected/a8b19c9c-11e2-4427-8229-86457df5ebfe-kube-api-access-vkgls\") pod \"a8b19c9c-11e2-4427-8229-86457df5ebfe\" (UID: \"a8b19c9c-11e2-4427-8229-86457df5ebfe\") "
Nov 03 03:12:50.785268 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:50.783292    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b19c9c-11e2-4427-8229-86457df5ebfe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b19c9c-11e2-4427-8229-86457df5ebfe-openshift-service-ca\") pod \"a8b19c9c-11e2-4427-8229-86457df5ebfe\" (UID: \"a8b19c9c-11e2-4427-8229-86457df5ebfe\") "
Nov 03 03:12:50.785268 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:50.783390    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b19c9c-11e2-4427-8229-86457df5ebfe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b19c9c-11e2-4427-8229-86457df5ebfe-metrics-certs\") pod \"a8b19c9c-11e2-4427-8229-86457df5ebfe\" (UID: \"a8b19c9c-11e2-4427-8229-86457df5ebfe\") "
Nov 03 03:12:50.785268 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:50.783494    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b19c9c-11e2-4427-8229-86457df5ebfe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8b19c9c-11e2-4427-8229-86457df5ebfe-maas-api-tls\") pod \"a8b19c9c-11e2-4427-8229-86457df5ebfe\" (UID: \"a8b19c9c-11e2-4427-8229-86457df5ebfe\") "
Nov 03 03:12:50.792848 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:50.792807    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b19c9c-11e2-4427-8229-86457df5ebfe-maas-api-tls" pod "a8b19c9c-11e2-4427-8229-86457df5ebfe" (UID: "a8b19c9c-11e2-4427-8229-86457df5ebfe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:12:50.796684 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:50.796569    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b19c9c-11e2-4427-8229-86457df5ebfe-kube-api-access-vkgls" pod "a8b19c9c-11e2-4427-8229-86457df5ebfe" (UID: "a8b19c9c-11e2-4427-8229-86457df5ebfe"). InnerVolumeSpecName "kube-api-access-vkgls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:12:50.797457 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:50.797425    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b19c9c-11e2-4427-8229-86457df5ebfe-openshift-service-ca" pod "a8b19c9c-11e2-4427-8229-86457df5ebfe" (UID: "a8b19c9c-11e2-4427-8229-86457df5ebfe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:12:50.802150 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:50.802118    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b19c9c-11e2-4427-8229-86457df5ebfe-metrics-certs" pod "a8b19c9c-11e2-4427-8229-86457df5ebfe" (UID: "a8b19c9c-11e2-4427-8229-86457df5ebfe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:12:50.884544 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:50.884505    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8b19c9c-11e2-4427-8229-86457df5ebfe-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:12:50.884544 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:50.884543    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b19c9c-11e2-4427-8229-86457df5ebfe-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:12:50.884789 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:50.884557    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8b19c9c-11e2-4427-8229-86457df5ebfe-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:12:50.884789 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:50.884570    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkgls\" (UniqueName: \"kubernetes.io/projected/a8b19c9c-11e2-4427-8229-86457df5ebfe-kube-api-access-vkgls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:12:51.253075 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:51.253040    5005 generic.go:356] "Generic (PLEG): container finished" podID="a8b19c9c-11e2-4427-8229-86457df5ebfe" containerID="6b040fae38d603bcbb5af0a634459b8d45f0e55e7988acf55c3ae3ccb828250b" exitCode=0
Nov 03 03:12:51.253375 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:51.253111    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"
Nov 03 03:12:51.253375 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:51.253123    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x" event={"ID":"a8b19c9c-11e2-4427-8229-86457df5ebfe","Type":"ContainerDied","Data":"6b040fae38d603bcbb5af0a634459b8d45f0e55e7988acf55c3ae3ccb828250b"}
Nov 03 03:12:51.253375 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:51.253177    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x" event={"ID":"a8b19c9c-11e2-4427-8229-86457df5ebfe","Type":"ContainerDied","Data":"1ffbe088e49c17b7952539c9e1b4e4afd61d9ff20a9d97ec4fdbb89b4eedb01e"}
Nov 03 03:12:51.253375 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:51.253200    5005 scope.go:122] "RemoveContainer" containerID="6b040fae38d603bcbb5af0a634459b8d45f0e55e7988acf55c3ae3ccb828250b"
Nov 03 03:12:51.262767 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:51.262749    5005 scope.go:122] "RemoveContainer" containerID="6b040fae38d603bcbb5af0a634459b8d45f0e55e7988acf55c3ae3ccb828250b"
Nov 03 03:12:51.263095 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:12:51.263067    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b040fae38d603bcbb5af0a634459b8d45f0e55e7988acf55c3ae3ccb828250b\": container with ID starting with 6b040fae38d603bcbb5af0a634459b8d45f0e55e7988acf55c3ae3ccb828250b not found: ID does not exist" containerID="6b040fae38d603bcbb5af0a634459b8d45f0e55e7988acf55c3ae3ccb828250b"
Nov 03 03:12:51.263187 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:51.263105    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b040fae38d603bcbb5af0a634459b8d45f0e55e7988acf55c3ae3ccb828250b"} err="failed to get container status \"6b040fae38d603bcbb5af0a634459b8d45f0e55e7988acf55c3ae3ccb828250b\": rpc error: code = NotFound desc = could not find container \"6b040fae38d603bcbb5af0a634459b8d45f0e55e7988acf55c3ae3ccb828250b\": container with ID starting with 6b040fae38d603bcbb5af0a634459b8d45f0e55e7988acf55c3ae3ccb828250b not found: ID does not exist"
Nov 03 03:12:51.406768 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:51.406628    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"]
Nov 03 03:12:51.453598 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:51.453558    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-78b5c5-d8b4464ff-z6w4x"]
Nov 03 03:12:53.112122 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:12:53.112035    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b19c9c-11e2-4427-8229-86457df5ebfe" path="/var/lib/kubelet/pods/a8b19c9c-11e2-4427-8229-86457df5ebfe/volumes"
Nov 03 03:13:00.714255 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:13:00.714215    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:13:01.587095 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:01.587057    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"]
Nov 03 03:13:11.356191 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:11.356153    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"]
Nov 03 03:13:11.356656 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:11.356417    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4" podUID="41a24e80-e0db-4bee-a1e8-eec710e6dd79" containerName="maas-api" containerID="cri-o://2e26f39880f0cfa094282c101c7f6277d436fa3a97190b7d81b33a23e152f36c" gracePeriod=30
Nov 03 03:13:11.628816 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:11.628787    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"
Nov 03 03:13:11.742473 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:11.742441    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41a24e80-e0db-4bee-a1e8-eec710e6dd79-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a24e80-e0db-4bee-a1e8-eec710e6dd79-openshift-service-ca\") pod \"41a24e80-e0db-4bee-a1e8-eec710e6dd79\" (UID: \"41a24e80-e0db-4bee-a1e8-eec710e6dd79\") "
Nov 03 03:13:11.742748 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:11.742730    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41a24e80-e0db-4bee-a1e8-eec710e6dd79-metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a24e80-e0db-4bee-a1e8-eec710e6dd79-metrics-certs\") pod \"41a24e80-e0db-4bee-a1e8-eec710e6dd79\" (UID: \"41a24e80-e0db-4bee-a1e8-eec710e6dd79\") "
Nov 03 03:13:11.743405 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:11.743382    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41a24e80-e0db-4bee-a1e8-eec710e6dd79-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a24e80-e0db-4bee-a1e8-eec710e6dd79-maas-api-tls\") pod \"41a24e80-e0db-4bee-a1e8-eec710e6dd79\" (UID: \"41a24e80-e0db-4bee-a1e8-eec710e6dd79\") "
Nov 03 03:13:11.743913 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:11.742865    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a24e80-e0db-4bee-a1e8-eec710e6dd79-openshift-service-ca" pod "41a24e80-e0db-4bee-a1e8-eec710e6dd79" (UID: "41a24e80-e0db-4bee-a1e8-eec710e6dd79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:13:11.744130 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:11.744108    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a24e80-e0db-4bee-a1e8-eec710e6dd79-kube-api-access-ps27l\" (UniqueName: \"kubernetes.io/projected/41a24e80-e0db-4bee-a1e8-eec710e6dd79-kube-api-access-ps27l\") pod \"41a24e80-e0db-4bee-a1e8-eec710e6dd79\" (UID: \"41a24e80-e0db-4bee-a1e8-eec710e6dd79\") "
Nov 03 03:13:11.744645 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:11.744626    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a24e80-e0db-4bee-a1e8-eec710e6dd79-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:11.750168 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:11.749774    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a24e80-e0db-4bee-a1e8-eec710e6dd79-kube-api-access-ps27l" pod "41a24e80-e0db-4bee-a1e8-eec710e6dd79" (UID: "41a24e80-e0db-4bee-a1e8-eec710e6dd79"). InnerVolumeSpecName "kube-api-access-ps27l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:13:11.757809 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:11.756255    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41a24e80-e0db-4bee-a1e8-eec710e6dd79-maas-api-tls" pod "41a24e80-e0db-4bee-a1e8-eec710e6dd79" (UID: "41a24e80-e0db-4bee-a1e8-eec710e6dd79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:13:11.759034 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:11.758656    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41a24e80-e0db-4bee-a1e8-eec710e6dd79-metrics-certs" pod "41a24e80-e0db-4bee-a1e8-eec710e6dd79" (UID: "41a24e80-e0db-4bee-a1e8-eec710e6dd79"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:13:11.845911 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:11.845877    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a24e80-e0db-4bee-a1e8-eec710e6dd79-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:11.845911 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:11.845909    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps27l\" (UniqueName: \"kubernetes.io/projected/41a24e80-e0db-4bee-a1e8-eec710e6dd79-kube-api-access-ps27l\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:11.845911 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:11.845920    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a24e80-e0db-4bee-a1e8-eec710e6dd79-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:12.386521 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:12.386488    5005 generic.go:356] "Generic (PLEG): container finished" podID="41a24e80-e0db-4bee-a1e8-eec710e6dd79" containerID="2e26f39880f0cfa094282c101c7f6277d436fa3a97190b7d81b33a23e152f36c" exitCode=0
Nov 03 03:13:12.387020 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:12.386530    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"
Nov 03 03:13:12.387020 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:12.386562    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4" event={"ID":"41a24e80-e0db-4bee-a1e8-eec710e6dd79","Type":"ContainerDied","Data":"2e26f39880f0cfa094282c101c7f6277d436fa3a97190b7d81b33a23e152f36c"}
Nov 03 03:13:12.387020 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:12.386613    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4" event={"ID":"41a24e80-e0db-4bee-a1e8-eec710e6dd79","Type":"ContainerDied","Data":"adbae7e70527c3bcdab0dc19cd05d39aed86fac00fc919aceea2adc31cece837"}
Nov 03 03:13:12.387020 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:12.386634    5005 scope.go:122] "RemoveContainer" containerID="2e26f39880f0cfa094282c101c7f6277d436fa3a97190b7d81b33a23e152f36c"
Nov 03 03:13:12.397016 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:12.396996    5005 scope.go:122] "RemoveContainer" containerID="2e26f39880f0cfa094282c101c7f6277d436fa3a97190b7d81b33a23e152f36c"
Nov 03 03:13:12.397309 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:13:12.397287    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e26f39880f0cfa094282c101c7f6277d436fa3a97190b7d81b33a23e152f36c\": container with ID starting with 2e26f39880f0cfa094282c101c7f6277d436fa3a97190b7d81b33a23e152f36c not found: ID does not exist" containerID="2e26f39880f0cfa094282c101c7f6277d436fa3a97190b7d81b33a23e152f36c"
Nov 03 03:13:12.397372 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:12.397323    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e26f39880f0cfa094282c101c7f6277d436fa3a97190b7d81b33a23e152f36c"} err="failed to get container status \"2e26f39880f0cfa094282c101c7f6277d436fa3a97190b7d81b33a23e152f36c\": rpc error: code = NotFound desc = could not find container \"2e26f39880f0cfa094282c101c7f6277d436fa3a97190b7d81b33a23e152f36c\": container with ID starting with 2e26f39880f0cfa094282c101c7f6277d436fa3a97190b7d81b33a23e152f36c not found: ID does not exist"
Nov 03 03:13:12.540158 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:12.540121    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"]
Nov 03 03:13:12.586542 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:12.586502    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f7d9bb-5ffbd9c56d-7hxj4"]
Nov 03 03:13:13.111443 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:13.111409    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41a24e80-e0db-4bee-a1e8-eec710e6dd79" path="/var/lib/kubelet/pods/41a24e80-e0db-4bee-a1e8-eec710e6dd79/volumes"
Nov 03 03:13:16.607441 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:16.607377    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd" podUID="fd71fd74-9581-4c83-8954-55462e1d4438" containerName="main" containerID="cri-o://0bbc2b454345e7a7ee8be0b9f2425976497a4293a9426a3fa2cd35a26a2b3ef0" gracePeriod=45
Nov 03 03:13:16.875069 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:16.875005    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 03:13:16.917970 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:16.917938    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-home\" (UniqueName: \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-home\") pod \"fd71fd74-9581-4c83-8954-55462e1d4438\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") "
Nov 03 03:13:16.918163 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:16.917995    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd71fd74-9581-4c83-8954-55462e1d4438-kube-api-access-spg95\" (UniqueName: \"kubernetes.io/projected/fd71fd74-9581-4c83-8954-55462e1d4438-kube-api-access-spg95\") pod \"fd71fd74-9581-4c83-8954-55462e1d4438\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") "
Nov 03 03:13:16.918163 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:16.918054    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-model-cache\") pod \"fd71fd74-9581-4c83-8954-55462e1d4438\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") "
Nov 03 03:13:16.918163 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:16.918101    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-dshm\") pod \"fd71fd74-9581-4c83-8954-55462e1d4438\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") "
Nov 03 03:13:16.918163 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:16.918127    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-tmp-dir\") pod \"fd71fd74-9581-4c83-8954-55462e1d4438\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") "
Nov 03 03:13:16.918359 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:16.918205    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd71fd74-9581-4c83-8954-55462e1d4438-tls-certs\" (UniqueName: \"kubernetes.io/secret/fd71fd74-9581-4c83-8954-55462e1d4438-tls-certs\") pod \"fd71fd74-9581-4c83-8954-55462e1d4438\" (UID: \"fd71fd74-9581-4c83-8954-55462e1d4438\") "
Nov 03 03:13:16.918359 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:16.918254    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-home" pod "fd71fd74-9581-4c83-8954-55462e1d4438" (UID: "fd71fd74-9581-4c83-8954-55462e1d4438"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:13:16.918457 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:16.918410    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-model-cache" pod "fd71fd74-9581-4c83-8954-55462e1d4438" (UID: "fd71fd74-9581-4c83-8954-55462e1d4438"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:13:16.918504 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:16.918447    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-tmp-dir" pod "fd71fd74-9581-4c83-8954-55462e1d4438" (UID: "fd71fd74-9581-4c83-8954-55462e1d4438"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:13:16.918646 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:16.918628    5005 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-home\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:16.918743 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:16.918652    5005 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-model-cache\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:16.918743 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:16.918664    5005 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-tmp-dir\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:16.920362 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:16.920339    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd71fd74-9581-4c83-8954-55462e1d4438-kube-api-access-spg95" pod "fd71fd74-9581-4c83-8954-55462e1d4438" (UID: "fd71fd74-9581-4c83-8954-55462e1d4438"). InnerVolumeSpecName "kube-api-access-spg95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:13:16.920472 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:16.920365    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd71fd74-9581-4c83-8954-55462e1d4438-tls-certs" pod "fd71fd74-9581-4c83-8954-55462e1d4438" (UID: "fd71fd74-9581-4c83-8954-55462e1d4438"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:13:16.920472 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:16.920372    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-dshm" pod "fd71fd74-9581-4c83-8954-55462e1d4438" (UID: "fd71fd74-9581-4c83-8954-55462e1d4438"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:13:17.020463 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:17.020425    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spg95\" (UniqueName: \"kubernetes.io/projected/fd71fd74-9581-4c83-8954-55462e1d4438-kube-api-access-spg95\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:17.020463 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:17.020466    5005 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd71fd74-9581-4c83-8954-55462e1d4438-dshm\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:17.020725 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:17.020481    5005 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd71fd74-9581-4c83-8954-55462e1d4438-tls-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:17.421570 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:17.421534    5005 generic.go:356] "Generic (PLEG): container finished" podID="fd71fd74-9581-4c83-8954-55462e1d4438" containerID="0bbc2b454345e7a7ee8be0b9f2425976497a4293a9426a3fa2cd35a26a2b3ef0" exitCode=0
Nov 03 03:13:17.421796 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:17.421575    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"
Nov 03 03:13:17.421796 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:17.421580    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd" event={"ID":"fd71fd74-9581-4c83-8954-55462e1d4438","Type":"ContainerDied","Data":"0bbc2b454345e7a7ee8be0b9f2425976497a4293a9426a3fa2cd35a26a2b3ef0"}
Nov 03 03:13:17.421796 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:17.421613    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd" event={"ID":"fd71fd74-9581-4c83-8954-55462e1d4438","Type":"ContainerDied","Data":"2ff2302c5181f74251bfd7c120d54d0455414fde005017ba1e1031ec492e5f16"}
Nov 03 03:13:17.421796 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:17.421630    5005 scope.go:122] "RemoveContainer" containerID="0bbc2b454345e7a7ee8be0b9f2425976497a4293a9426a3fa2cd35a26a2b3ef0"
Nov 03 03:13:17.430974 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:17.430934    5005 scope.go:122] "RemoveContainer" containerID="0bbc2b454345e7a7ee8be0b9f2425976497a4293a9426a3fa2cd35a26a2b3ef0"
Nov 03 03:13:17.431243 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:13:17.431224    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bbc2b454345e7a7ee8be0b9f2425976497a4293a9426a3fa2cd35a26a2b3ef0\": container with ID starting with 0bbc2b454345e7a7ee8be0b9f2425976497a4293a9426a3fa2cd35a26a2b3ef0 not found: ID does not exist" containerID="0bbc2b454345e7a7ee8be0b9f2425976497a4293a9426a3fa2cd35a26a2b3ef0"
Nov 03 03:13:17.431295 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:17.431252    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bbc2b454345e7a7ee8be0b9f2425976497a4293a9426a3fa2cd35a26a2b3ef0"} err="failed to get container status \"0bbc2b454345e7a7ee8be0b9f2425976497a4293a9426a3fa2cd35a26a2b3ef0\": rpc error: code = NotFound desc = could not find container \"0bbc2b454345e7a7ee8be0b9f2425976497a4293a9426a3fa2cd35a26a2b3ef0\": container with ID starting with 0bbc2b454345e7a7ee8be0b9f2425976497a4293a9426a3fa2cd35a26a2b3ef0 not found: ID does not exist"
Nov 03 03:13:17.575606 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:17.575568    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"]
Nov 03 03:13:17.623718 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:17.623673    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-1bccbd/simulator-w1-1bccbd-kserve-795b7bc77c-skmcd"]
Nov 03 03:13:19.109856 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:19.109823    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd71fd74-9581-4c83-8954-55462e1d4438" path="/var/lib/kubelet/pods/fd71fd74-9581-4c83-8954-55462e1d4438/volumes"
Nov 03 03:13:30.714431 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:13:30.714388    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:13:30.848036 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:30.847994    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"]
Nov 03 03:13:30.848653 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:30.848601    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w" podUID="700d764b-0150-4711-aa7c-718170d233d5" containerName="istio-proxy" containerID="cri-o://a476974720c576cb6b85b386a528ca2c0c5a97cc903169c280b08bf4a18ee733" gracePeriod=30
Nov 03 03:13:32.058108 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:32.058064    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7fn2w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 03:13:32.058489 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:32.058137    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w" podUID="700d764b-0150-4711-aa7c-718170d233d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 03:13:36.116658 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.116628    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 03:13:36.240767 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.240671    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-workload-socket\") pod \"700d764b-0150-4711-aa7c-718170d233d5\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") "
Nov 03 03:13:36.240767 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.240744    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-workload-certs\") pod \"700d764b-0150-4711-aa7c-718170d233d5\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") "
Nov 03 03:13:36.241011 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.240774    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700d764b-0150-4711-aa7c-718170d233d5-istio-token\" (UniqueName: \"kubernetes.io/projected/700d764b-0150-4711-aa7c-718170d233d5-istio-token\") pod \"700d764b-0150-4711-aa7c-718170d233d5\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") "
Nov 03 03:13:36.241011 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.240841    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-istio-envoy\") pod \"700d764b-0150-4711-aa7c-718170d233d5\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") "
Nov 03 03:13:36.241011 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.240863    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-credential-socket\") pod \"700d764b-0150-4711-aa7c-718170d233d5\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") "
Nov 03 03:13:36.241011 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.240940    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-istio-data\") pod \"700d764b-0150-4711-aa7c-718170d233d5\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") "
Nov 03 03:13:36.241011 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.240974    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/700d764b-0150-4711-aa7c-718170d233d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700d764b-0150-4711-aa7c-718170d233d5-istio-podinfo\") pod \"700d764b-0150-4711-aa7c-718170d233d5\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") "
Nov 03 03:13:36.241011 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.240975    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-workload-socket" pod "700d764b-0150-4711-aa7c-718170d233d5" (UID: "700d764b-0150-4711-aa7c-718170d233d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:13:36.241011 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.241006    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700d764b-0150-4711-aa7c-718170d233d5-kube-api-access-4kvm5\" (UniqueName: \"kubernetes.io/projected/700d764b-0150-4711-aa7c-718170d233d5-kube-api-access-4kvm5\") pod \"700d764b-0150-4711-aa7c-718170d233d5\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") "
Nov 03 03:13:36.241350 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.241044    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/700d764b-0150-4711-aa7c-718170d233d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700d764b-0150-4711-aa7c-718170d233d5-istiod-ca-cert\") pod \"700d764b-0150-4711-aa7c-718170d233d5\" (UID: \"700d764b-0150-4711-aa7c-718170d233d5\") "
Nov 03 03:13:36.241350 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.241060    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-workload-certs" pod "700d764b-0150-4711-aa7c-718170d233d5" (UID: "700d764b-0150-4711-aa7c-718170d233d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:13:36.241350 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.241205    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-credential-socket" pod "700d764b-0150-4711-aa7c-718170d233d5" (UID: "700d764b-0150-4711-aa7c-718170d233d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:13:36.241350 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.241225    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-istio-data" pod "700d764b-0150-4711-aa7c-718170d233d5" (UID: "700d764b-0150-4711-aa7c-718170d233d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:13:36.241557 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.241476    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:36.241557 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.241507    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:36.241557 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.241523    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:36.241557 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.241536    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:36.241557 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.241484    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/700d764b-0150-4711-aa7c-718170d233d5-istiod-ca-cert" pod "700d764b-0150-4711-aa7c-718170d233d5" (UID: "700d764b-0150-4711-aa7c-718170d233d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:13:36.243247 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.243218    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700d764b-0150-4711-aa7c-718170d233d5-istio-token" pod "700d764b-0150-4711-aa7c-718170d233d5" (UID: "700d764b-0150-4711-aa7c-718170d233d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:13:36.243247 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.243232    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700d764b-0150-4711-aa7c-718170d233d5-kube-api-access-4kvm5" pod "700d764b-0150-4711-aa7c-718170d233d5" (UID: "700d764b-0150-4711-aa7c-718170d233d5"). InnerVolumeSpecName "kube-api-access-4kvm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:13:36.243247 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.243239    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-istio-envoy" pod "700d764b-0150-4711-aa7c-718170d233d5" (UID: "700d764b-0150-4711-aa7c-718170d233d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:13:36.243437 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.243338    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/700d764b-0150-4711-aa7c-718170d233d5-istio-podinfo" pod "700d764b-0150-4711-aa7c-718170d233d5" (UID: "700d764b-0150-4711-aa7c-718170d233d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:13:36.342675 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.342639    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700d764b-0150-4711-aa7c-718170d233d5-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:36.342675 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.342669    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kvm5\" (UniqueName: \"kubernetes.io/projected/700d764b-0150-4711-aa7c-718170d233d5-kube-api-access-4kvm5\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:36.342675 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.342679    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700d764b-0150-4711-aa7c-718170d233d5-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:36.342964 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.342689    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700d764b-0150-4711-aa7c-718170d233d5-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:36.342964 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.342726    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700d764b-0150-4711-aa7c-718170d233d5-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:36.531219 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.531181    5005 generic.go:356] "Generic (PLEG): container finished" podID="700d764b-0150-4711-aa7c-718170d233d5" containerID="a476974720c576cb6b85b386a528ca2c0c5a97cc903169c280b08bf4a18ee733" exitCode=0
Nov 03 03:13:36.531412 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.531232    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"
Nov 03 03:13:36.531412 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.531282    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w" event={"ID":"700d764b-0150-4711-aa7c-718170d233d5","Type":"ContainerDied","Data":"a476974720c576cb6b85b386a528ca2c0c5a97cc903169c280b08bf4a18ee733"}
Nov 03 03:13:36.531412 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.531340    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w" event={"ID":"700d764b-0150-4711-aa7c-718170d233d5","Type":"ContainerDied","Data":"201ca749f646501bdf57f2abf737c6310de8609540b1688896098bd9fad9ae23"}
Nov 03 03:13:36.531412 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.531368    5005 scope.go:122] "RemoveContainer" containerID="a476974720c576cb6b85b386a528ca2c0c5a97cc903169c280b08bf4a18ee733"
Nov 03 03:13:36.542393 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.542376    5005 scope.go:122] "RemoveContainer" containerID="a476974720c576cb6b85b386a528ca2c0c5a97cc903169c280b08bf4a18ee733"
Nov 03 03:13:36.542668 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:13:36.542652    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a476974720c576cb6b85b386a528ca2c0c5a97cc903169c280b08bf4a18ee733\": container with ID starting with a476974720c576cb6b85b386a528ca2c0c5a97cc903169c280b08bf4a18ee733 not found: ID does not exist" containerID="a476974720c576cb6b85b386a528ca2c0c5a97cc903169c280b08bf4a18ee733"
Nov 03 03:13:36.542761 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.542675    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a476974720c576cb6b85b386a528ca2c0c5a97cc903169c280b08bf4a18ee733"} err="failed to get container status \"a476974720c576cb6b85b386a528ca2c0c5a97cc903169c280b08bf4a18ee733\": rpc error: code = NotFound desc = could not find container \"a476974720c576cb6b85b386a528ca2c0c5a97cc903169c280b08bf4a18ee733\": container with ID starting with a476974720c576cb6b85b386a528ca2c0c5a97cc903169c280b08bf4a18ee733 not found: ID does not exist"
Nov 03 03:13:36.687191 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.687159    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"]
Nov 03 03:13:36.690524 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:36.690496    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7fn2w"]
Nov 03 03:13:37.109377 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:37.109344    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="700d764b-0150-4711-aa7c-718170d233d5" path="/var/lib/kubelet/pods/700d764b-0150-4711-aa7c-718170d233d5/volumes"
Nov 03 03:13:39.151502 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.151470    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"]
Nov 03 03:13:39.151953 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.151675    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j" podUID="532d5d68-bc86-491b-978b-97f9cf54580a" containerName="maas-api" containerID="cri-o://dbb9867a9bed6184188617785575b72a86e40235e37d56ead9ebb319bb6a6968" gracePeriod=30
Nov 03 03:13:39.439973 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.439945    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"
Nov 03 03:13:39.478100 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.478066    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/532d5d68-bc86-491b-978b-97f9cf54580a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/532d5d68-bc86-491b-978b-97f9cf54580a-maas-api-tls\") pod \"532d5d68-bc86-491b-978b-97f9cf54580a\" (UID: \"532d5d68-bc86-491b-978b-97f9cf54580a\") "
Nov 03 03:13:39.478267 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.478105    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/532d5d68-bc86-491b-978b-97f9cf54580a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/532d5d68-bc86-491b-978b-97f9cf54580a-openshift-service-ca\") pod \"532d5d68-bc86-491b-978b-97f9cf54580a\" (UID: \"532d5d68-bc86-491b-978b-97f9cf54580a\") "
Nov 03 03:13:39.478267 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.478178    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/532d5d68-bc86-491b-978b-97f9cf54580a-kube-api-access-np68m\" (UniqueName: \"kubernetes.io/projected/532d5d68-bc86-491b-978b-97f9cf54580a-kube-api-access-np68m\") pod \"532d5d68-bc86-491b-978b-97f9cf54580a\" (UID: \"532d5d68-bc86-491b-978b-97f9cf54580a\") "
Nov 03 03:13:39.478267 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.478254    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/532d5d68-bc86-491b-978b-97f9cf54580a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/532d5d68-bc86-491b-978b-97f9cf54580a-metrics-certs\") pod \"532d5d68-bc86-491b-978b-97f9cf54580a\" (UID: \"532d5d68-bc86-491b-978b-97f9cf54580a\") "
Nov 03 03:13:39.478510 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.478488    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/532d5d68-bc86-491b-978b-97f9cf54580a-openshift-service-ca" pod "532d5d68-bc86-491b-978b-97f9cf54580a" (UID: "532d5d68-bc86-491b-978b-97f9cf54580a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:13:39.478718 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.478676    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/532d5d68-bc86-491b-978b-97f9cf54580a-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:39.480391 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.480370    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/532d5d68-bc86-491b-978b-97f9cf54580a-maas-api-tls" pod "532d5d68-bc86-491b-978b-97f9cf54580a" (UID: "532d5d68-bc86-491b-978b-97f9cf54580a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:13:39.480492 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.480391    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/532d5d68-bc86-491b-978b-97f9cf54580a-metrics-certs" pod "532d5d68-bc86-491b-978b-97f9cf54580a" (UID: "532d5d68-bc86-491b-978b-97f9cf54580a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:13:39.480534 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.480490    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532d5d68-bc86-491b-978b-97f9cf54580a-kube-api-access-np68m" pod "532d5d68-bc86-491b-978b-97f9cf54580a" (UID: "532d5d68-bc86-491b-978b-97f9cf54580a"). InnerVolumeSpecName "kube-api-access-np68m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:13:39.555015 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.554953    5005 generic.go:356] "Generic (PLEG): container finished" podID="532d5d68-bc86-491b-978b-97f9cf54580a" containerID="dbb9867a9bed6184188617785575b72a86e40235e37d56ead9ebb319bb6a6968" exitCode=0
Nov 03 03:13:39.555187 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.555036    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"
Nov 03 03:13:39.555187 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.555039    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j" event={"ID":"532d5d68-bc86-491b-978b-97f9cf54580a","Type":"ContainerDied","Data":"dbb9867a9bed6184188617785575b72a86e40235e37d56ead9ebb319bb6a6968"}
Nov 03 03:13:39.555187 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.555124    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j" event={"ID":"532d5d68-bc86-491b-978b-97f9cf54580a","Type":"ContainerDied","Data":"2d97df12d7689a56380cf493fa2681dd4a204e9ac265e104fc215f6de80293aa"}
Nov 03 03:13:39.555187 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.555165    5005 scope.go:122] "RemoveContainer" containerID="dbb9867a9bed6184188617785575b72a86e40235e37d56ead9ebb319bb6a6968"
Nov 03 03:13:39.568307 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.568271    5005 scope.go:122] "RemoveContainer" containerID="dbb9867a9bed6184188617785575b72a86e40235e37d56ead9ebb319bb6a6968"
Nov 03 03:13:39.570224 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:13:39.570194    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbb9867a9bed6184188617785575b72a86e40235e37d56ead9ebb319bb6a6968\": container with ID starting with dbb9867a9bed6184188617785575b72a86e40235e37d56ead9ebb319bb6a6968 not found: ID does not exist" containerID="dbb9867a9bed6184188617785575b72a86e40235e37d56ead9ebb319bb6a6968"
Nov 03 03:13:39.570325 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.570238    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbb9867a9bed6184188617785575b72a86e40235e37d56ead9ebb319bb6a6968"} err="failed to get container status \"dbb9867a9bed6184188617785575b72a86e40235e37d56ead9ebb319bb6a6968\": rpc error: code = NotFound desc = could not find container \"dbb9867a9bed6184188617785575b72a86e40235e37d56ead9ebb319bb6a6968\": container with ID starting with dbb9867a9bed6184188617785575b72a86e40235e37d56ead9ebb319bb6a6968 not found: ID does not exist"
Nov 03 03:13:39.580376 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.580347    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np68m\" (UniqueName: \"kubernetes.io/projected/532d5d68-bc86-491b-978b-97f9cf54580a-kube-api-access-np68m\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:39.580480 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.580381    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/532d5d68-bc86-491b-978b-97f9cf54580a-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:39.580480 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.580397    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/532d5d68-bc86-491b-978b-97f9cf54580a-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:39.708654 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.708571    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"]
Nov 03 03:13:39.713962 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:39.713932    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-1bccbd-69f6d78578-vql4j"]
Nov 03 03:13:41.108481 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:41.108444    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="532d5d68-bc86-491b-978b-97f9cf54580a" path="/var/lib/kubelet/pods/532d5d68-bc86-491b-978b-97f9cf54580a/volumes"
Nov 03 03:13:50.698314 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:50.698259    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"]
Nov 03 03:13:50.698880 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:50.698529    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68" podUID="f0019a0f-0716-4bd2-be7b-f96ff62bf7d1" containerName="istio-proxy" containerID="cri-o://d76e0d9841f3f47eb2179c3f908f649313a9c83dcc43f6ca4ec97a1ef489d257" gracePeriod=30
Nov 03 03:13:55.968494 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:55.968431    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 03:13:56.066882 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.066847    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istiod-ca-cert\") pod \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") "
Nov 03 03:13:56.067097 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.066915    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-envoy\") pod \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") "
Nov 03 03:13:56.067097 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.066959    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-credential-socket\") pod \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") "
Nov 03 03:13:56.067097 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.067006    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-workload-socket\") pod \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") "
Nov 03 03:13:56.067097 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.067074    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-workload-certs\") pod \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") "
Nov 03 03:13:56.067322 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.067112    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-token\" (UniqueName: \"kubernetes.io/projected/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-token\") pod \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") "
Nov 03 03:13:56.067322 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.067136    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-kube-api-access-t5zwr\" (UniqueName: \"kubernetes.io/projected/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-kube-api-access-t5zwr\") pod \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") "
Nov 03 03:13:56.067322 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.067192    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-data\") pod \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") "
Nov 03 03:13:56.067322 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.067266    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-credential-socket" pod "f0019a0f-0716-4bd2-be7b-f96ff62bf7d1" (UID: "f0019a0f-0716-4bd2-be7b-f96ff62bf7d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:13:56.067322 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.067302    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-podinfo\") pod \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\" (UID: \"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1\") "
Nov 03 03:13:56.067619 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.067541    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-data" pod "f0019a0f-0716-4bd2-be7b-f96ff62bf7d1" (UID: "f0019a0f-0716-4bd2-be7b-f96ff62bf7d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:13:56.067619 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.067558    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-workload-certs" pod "f0019a0f-0716-4bd2-be7b-f96ff62bf7d1" (UID: "f0019a0f-0716-4bd2-be7b-f96ff62bf7d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:13:56.067731 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.067630    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-workload-socket" pod "f0019a0f-0716-4bd2-be7b-f96ff62bf7d1" (UID: "f0019a0f-0716-4bd2-be7b-f96ff62bf7d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:13:56.067731 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.067677    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istiod-ca-cert" pod "f0019a0f-0716-4bd2-be7b-f96ff62bf7d1" (UID: "f0019a0f-0716-4bd2-be7b-f96ff62bf7d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:13:56.067872 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.067846    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:56.067923 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.067877    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:56.067923 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.067893    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:56.067923 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.067907    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:56.067923 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.067921    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:56.069388 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.069366    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-kube-api-access-t5zwr" pod "f0019a0f-0716-4bd2-be7b-f96ff62bf7d1" (UID: "f0019a0f-0716-4bd2-be7b-f96ff62bf7d1"). InnerVolumeSpecName "kube-api-access-t5zwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:13:56.069539 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.069517    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-podinfo" pod "f0019a0f-0716-4bd2-be7b-f96ff62bf7d1" (UID: "f0019a0f-0716-4bd2-be7b-f96ff62bf7d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:13:56.069607 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.069578    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-envoy" pod "f0019a0f-0716-4bd2-be7b-f96ff62bf7d1" (UID: "f0019a0f-0716-4bd2-be7b-f96ff62bf7d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:13:56.069809 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.069783    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-token" pod "f0019a0f-0716-4bd2-be7b-f96ff62bf7d1" (UID: "f0019a0f-0716-4bd2-be7b-f96ff62bf7d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:13:56.169391 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.169350    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:56.169391 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.169385    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:56.169391 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.169396    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5zwr\" (UniqueName: \"kubernetes.io/projected/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-kube-api-access-t5zwr\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:56.169628 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.169406    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:13:56.654172 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.654134    5005 generic.go:356] "Generic (PLEG): container finished" podID="f0019a0f-0716-4bd2-be7b-f96ff62bf7d1" containerID="d76e0d9841f3f47eb2179c3f908f649313a9c83dcc43f6ca4ec97a1ef489d257" exitCode=0
Nov 03 03:13:56.654361 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.654183    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"
Nov 03 03:13:56.654361 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.654216    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68" event={"ID":"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1","Type":"ContainerDied","Data":"d76e0d9841f3f47eb2179c3f908f649313a9c83dcc43f6ca4ec97a1ef489d257"}
Nov 03 03:13:56.654361 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.654269    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68" event={"ID":"f0019a0f-0716-4bd2-be7b-f96ff62bf7d1","Type":"ContainerDied","Data":"41508d2d1e215bedec11301081066c362885a79511e0811b8c3233f374530afc"}
Nov 03 03:13:56.654361 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.654291    5005 scope.go:122] "RemoveContainer" containerID="d76e0d9841f3f47eb2179c3f908f649313a9c83dcc43f6ca4ec97a1ef489d257"
Nov 03 03:13:56.664943 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.664921    5005 scope.go:122] "RemoveContainer" containerID="d76e0d9841f3f47eb2179c3f908f649313a9c83dcc43f6ca4ec97a1ef489d257"
Nov 03 03:13:56.665227 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:13:56.665209    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d76e0d9841f3f47eb2179c3f908f649313a9c83dcc43f6ca4ec97a1ef489d257\": container with ID starting with d76e0d9841f3f47eb2179c3f908f649313a9c83dcc43f6ca4ec97a1ef489d257 not found: ID does not exist" containerID="d76e0d9841f3f47eb2179c3f908f649313a9c83dcc43f6ca4ec97a1ef489d257"
Nov 03 03:13:56.665265 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.665237    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d76e0d9841f3f47eb2179c3f908f649313a9c83dcc43f6ca4ec97a1ef489d257"} err="failed to get container status \"d76e0d9841f3f47eb2179c3f908f649313a9c83dcc43f6ca4ec97a1ef489d257\": rpc error: code = NotFound desc = could not find container \"d76e0d9841f3f47eb2179c3f908f649313a9c83dcc43f6ca4ec97a1ef489d257\": container with ID starting with d76e0d9841f3f47eb2179c3f908f649313a9c83dcc43f6ca4ec97a1ef489d257 not found: ID does not exist"
Nov 03 03:13:56.809786 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.809748    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"]
Nov 03 03:13:56.844315 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:56.844276    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f735f4-openshift-default-7bdbb4545c-fcb68"]
Nov 03 03:13:57.109614 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:13:57.109576    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0019a0f-0716-4bd2-be7b-f96ff62bf7d1" path="/var/lib/kubelet/pods/f0019a0f-0716-4bd2-be7b-f96ff62bf7d1/volumes"
Nov 03 03:14:00.715108 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:14:00.715064    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:14:10.122936 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.122894    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"]
Nov 03 03:14:10.123532 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.123122    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9" podUID="5618cf05-a244-4531-9eca-ee70c484da3e" containerName="maas-api" containerID="cri-o://3d0ccbb607822f8f459c3a6e646c849dcef8feeebec3784c44058f4e6a058137" gracePeriod=30
Nov 03 03:14:10.406045 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.406016    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"
Nov 03 03:14:10.421572 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.421545    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5618cf05-a244-4531-9eca-ee70c484da3e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5618cf05-a244-4531-9eca-ee70c484da3e-metrics-certs\") pod \"5618cf05-a244-4531-9eca-ee70c484da3e\" (UID: \"5618cf05-a244-4531-9eca-ee70c484da3e\") "
Nov 03 03:14:10.421734 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.421599    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5618cf05-a244-4531-9eca-ee70c484da3e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5618cf05-a244-4531-9eca-ee70c484da3e-openshift-service-ca\") pod \"5618cf05-a244-4531-9eca-ee70c484da3e\" (UID: \"5618cf05-a244-4531-9eca-ee70c484da3e\") "
Nov 03 03:14:10.421734 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.421642    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5618cf05-a244-4531-9eca-ee70c484da3e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5618cf05-a244-4531-9eca-ee70c484da3e-maas-api-tls\") pod \"5618cf05-a244-4531-9eca-ee70c484da3e\" (UID: \"5618cf05-a244-4531-9eca-ee70c484da3e\") "
Nov 03 03:14:10.421734 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.421667    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5618cf05-a244-4531-9eca-ee70c484da3e-kube-api-access-lws4p\" (UniqueName: \"kubernetes.io/projected/5618cf05-a244-4531-9eca-ee70c484da3e-kube-api-access-lws4p\") pod \"5618cf05-a244-4531-9eca-ee70c484da3e\" (UID: \"5618cf05-a244-4531-9eca-ee70c484da3e\") "
Nov 03 03:14:10.422098 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.422062    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5618cf05-a244-4531-9eca-ee70c484da3e-openshift-service-ca" pod "5618cf05-a244-4531-9eca-ee70c484da3e" (UID: "5618cf05-a244-4531-9eca-ee70c484da3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:14:10.423849 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.423825    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5618cf05-a244-4531-9eca-ee70c484da3e-maas-api-tls" pod "5618cf05-a244-4531-9eca-ee70c484da3e" (UID: "5618cf05-a244-4531-9eca-ee70c484da3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:14:10.423945 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.423864    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5618cf05-a244-4531-9eca-ee70c484da3e-kube-api-access-lws4p" pod "5618cf05-a244-4531-9eca-ee70c484da3e" (UID: "5618cf05-a244-4531-9eca-ee70c484da3e"). InnerVolumeSpecName "kube-api-access-lws4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:14:10.424011 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.423985    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5618cf05-a244-4531-9eca-ee70c484da3e-metrics-certs" pod "5618cf05-a244-4531-9eca-ee70c484da3e" (UID: "5618cf05-a244-4531-9eca-ee70c484da3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:14:10.523810 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.523775    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5618cf05-a244-4531-9eca-ee70c484da3e-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:14:10.523810 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.523805    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5618cf05-a244-4531-9eca-ee70c484da3e-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:14:10.523810 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.523815    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lws4p\" (UniqueName: \"kubernetes.io/projected/5618cf05-a244-4531-9eca-ee70c484da3e-kube-api-access-lws4p\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:14:10.524154 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.523825    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5618cf05-a244-4531-9eca-ee70c484da3e-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:14:10.746786 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.746663    5005 generic.go:356] "Generic (PLEG): container finished" podID="5618cf05-a244-4531-9eca-ee70c484da3e" containerID="3d0ccbb607822f8f459c3a6e646c849dcef8feeebec3784c44058f4e6a058137" exitCode=0
Nov 03 03:14:10.746786 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.746734    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"
Nov 03 03:14:10.747008 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.746732    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9" event={"ID":"5618cf05-a244-4531-9eca-ee70c484da3e","Type":"ContainerDied","Data":"3d0ccbb607822f8f459c3a6e646c849dcef8feeebec3784c44058f4e6a058137"}
Nov 03 03:14:10.747008 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.746887    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9" event={"ID":"5618cf05-a244-4531-9eca-ee70c484da3e","Type":"ContainerDied","Data":"a32108c6e9bbc096bfe05c6a09bb50e2650a8df10bbaa8141996f201b7b5072c"}
Nov 03 03:14:10.747008 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.746919    5005 scope.go:122] "RemoveContainer" containerID="3d0ccbb607822f8f459c3a6e646c849dcef8feeebec3784c44058f4e6a058137"
Nov 03 03:14:10.757734 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.757683    5005 scope.go:122] "RemoveContainer" containerID="3d0ccbb607822f8f459c3a6e646c849dcef8feeebec3784c44058f4e6a058137"
Nov 03 03:14:10.758000 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:14:10.757982    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0ccbb607822f8f459c3a6e646c849dcef8feeebec3784c44058f4e6a058137\": container with ID starting with 3d0ccbb607822f8f459c3a6e646c849dcef8feeebec3784c44058f4e6a058137 not found: ID does not exist" containerID="3d0ccbb607822f8f459c3a6e646c849dcef8feeebec3784c44058f4e6a058137"
Nov 03 03:14:10.758043 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.758011    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0ccbb607822f8f459c3a6e646c849dcef8feeebec3784c44058f4e6a058137"} err="failed to get container status \"3d0ccbb607822f8f459c3a6e646c849dcef8feeebec3784c44058f4e6a058137\": rpc error: code = NotFound desc = could not find container \"3d0ccbb607822f8f459c3a6e646c849dcef8feeebec3784c44058f4e6a058137\": container with ID starting with 3d0ccbb607822f8f459c3a6e646c849dcef8feeebec3784c44058f4e6a058137 not found: ID does not exist"
Nov 03 03:14:10.901187 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.901155    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"]
Nov 03 03:14:10.947752 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:10.947722    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-957b33-6768c95dc-bw6f9"]
Nov 03 03:14:11.108679 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:11.108644    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5618cf05-a244-4531-9eca-ee70c484da3e" path="/var/lib/kubelet/pods/5618cf05-a244-4531-9eca-ee70c484da3e/volumes"
Nov 03 03:14:24.743884 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:24.743798    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"]
Nov 03 03:14:30.715405 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:14:30.715362    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:14:39.764303 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:39.764259    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm" podUID="b1f526d9-2fd8-4c8d-9175-732afdbf3570" containerName="main" containerID="cri-o://194e25547ee2ad1129d8f0cf1872e68d7729918aee728405d9adcea59477bdee" gracePeriod=45
Nov 03 03:14:39.936408 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:39.936378    5005 generic.go:356] "Generic (PLEG): container finished" podID="b1f526d9-2fd8-4c8d-9175-732afdbf3570" containerID="194e25547ee2ad1129d8f0cf1872e68d7729918aee728405d9adcea59477bdee" exitCode=0
Nov 03 03:14:39.936553 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:39.936454    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm" event={"ID":"b1f526d9-2fd8-4c8d-9175-732afdbf3570","Type":"ContainerDied","Data":"194e25547ee2ad1129d8f0cf1872e68d7729918aee728405d9adcea59477bdee"}
Nov 03 03:14:40.029369 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:40.029305    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 03:14:40.048598 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:40.048569    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1f526d9-2fd8-4c8d-9175-732afdbf3570-tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f526d9-2fd8-4c8d-9175-732afdbf3570-tls-certs\") pod \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") "
Nov 03 03:14:40.048767 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:40.048640    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-model-cache\") pod \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") "
Nov 03 03:14:40.048767 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:40.048723    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-dshm\") pod \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") "
Nov 03 03:14:40.048901 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:40.048821    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1f526d9-2fd8-4c8d-9175-732afdbf3570-kube-api-access-4tmr2\" (UniqueName: \"kubernetes.io/projected/b1f526d9-2fd8-4c8d-9175-732afdbf3570-kube-api-access-4tmr2\") pod \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") "
Nov 03 03:14:40.048901 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:40.048878    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-tmp-dir\") pod \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") "
Nov 03 03:14:40.049021 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:40.048904    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-home\" (UniqueName: \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-home\") pod \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\" (UID: \"b1f526d9-2fd8-4c8d-9175-732afdbf3570\") "
Nov 03 03:14:40.049021 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:40.048921    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-model-cache" pod "b1f526d9-2fd8-4c8d-9175-732afdbf3570" (UID: "b1f526d9-2fd8-4c8d-9175-732afdbf3570"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:14:40.049195 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:40.049170    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-home" pod "b1f526d9-2fd8-4c8d-9175-732afdbf3570" (UID: "b1f526d9-2fd8-4c8d-9175-732afdbf3570"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:14:40.049306 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:40.049192    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-tmp-dir" pod "b1f526d9-2fd8-4c8d-9175-732afdbf3570" (UID: "b1f526d9-2fd8-4c8d-9175-732afdbf3570"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:14:40.049449 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:40.049429    5005 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-model-cache\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:14:40.049512 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:40.049482    5005 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-tmp-dir\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:14:40.049512 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:40.049500    5005 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-home\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:14:40.051036 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:40.051011    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-dshm" pod "b1f526d9-2fd8-4c8d-9175-732afdbf3570" (UID: "b1f526d9-2fd8-4c8d-9175-732afdbf3570"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:14:40.051129 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:40.051047    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f526d9-2fd8-4c8d-9175-732afdbf3570-kube-api-access-4tmr2" pod "b1f526d9-2fd8-4c8d-9175-732afdbf3570" (UID: "b1f526d9-2fd8-4c8d-9175-732afdbf3570"). InnerVolumeSpecName "kube-api-access-4tmr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:14:40.051129 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:40.051063    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1f526d9-2fd8-4c8d-9175-732afdbf3570-tls-certs" pod "b1f526d9-2fd8-4c8d-9175-732afdbf3570" (UID: "b1f526d9-2fd8-4c8d-9175-732afdbf3570"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:14:40.150956 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:40.150909    5005 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f526d9-2fd8-4c8d-9175-732afdbf3570-tls-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:14:40.150956 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:40.150956    5005 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1f526d9-2fd8-4c8d-9175-732afdbf3570-dshm\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:14:40.151153 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:40.150971    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tmr2\" (UniqueName: \"kubernetes.io/projected/b1f526d9-2fd8-4c8d-9175-732afdbf3570-kube-api-access-4tmr2\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:14:40.944298 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:40.944265    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"
Nov 03 03:14:40.944797 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:40.944266    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm" event={"ID":"b1f526d9-2fd8-4c8d-9175-732afdbf3570","Type":"ContainerDied","Data":"cf3a1108cc5b9315e90bc3845bff75396bee009b9ebdeb9fa8a628b671058177"}
Nov 03 03:14:40.944797 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:40.944401    5005 scope.go:122] "RemoveContainer" containerID="194e25547ee2ad1129d8f0cf1872e68d7729918aee728405d9adcea59477bdee"
Nov 03 03:14:41.099549 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:41.099513    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"]
Nov 03 03:14:41.145675 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:41.145639    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-521f1f/distinct-2-w2-521f1f-kserve-896d88b95-bs4pm"]
Nov 03 03:14:43.109814 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:43.109775    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1f526d9-2fd8-4c8d-9175-732afdbf3570" path="/var/lib/kubelet/pods/b1f526d9-2fd8-4c8d-9175-732afdbf3570/volumes"
Nov 03 03:14:45.634727 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:45.634677    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"]
Nov 03 03:14:45.635164 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:45.634912    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw" podUID="81bdbd55-f2ba-4334-8c9f-42d61fea3b33" containerName="maas-api" containerID="cri-o://c33571d4623196ad2774f4015575df5ca45e881aac105368884d7bc40243b712" gracePeriod=30
Nov 03 03:14:45.780298 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:45.780248    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw" podUID="81bdbd55-f2ba-4334-8c9f-42d61fea3b33" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.70:8443/health\": dial tcp 10.132.0.70:8443: connect: connection refused"
Nov 03 03:14:45.912398 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:45.912365    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"
Nov 03 03:14:45.975351 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:45.975318    5005 generic.go:356] "Generic (PLEG): container finished" podID="81bdbd55-f2ba-4334-8c9f-42d61fea3b33" containerID="c33571d4623196ad2774f4015575df5ca45e881aac105368884d7bc40243b712" exitCode=0
Nov 03 03:14:45.975519 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:45.975370    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"
Nov 03 03:14:45.975519 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:45.975384    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw" event={"ID":"81bdbd55-f2ba-4334-8c9f-42d61fea3b33","Type":"ContainerDied","Data":"c33571d4623196ad2774f4015575df5ca45e881aac105368884d7bc40243b712"}
Nov 03 03:14:45.975519 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:45.975428    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw" event={"ID":"81bdbd55-f2ba-4334-8c9f-42d61fea3b33","Type":"ContainerDied","Data":"36aa2c0ea1157f4bd01b9dd1218c86e250cd48b6f88e19985f0d7ab7262f8644"}
Nov 03 03:14:45.975519 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:45.975450    5005 scope.go:122] "RemoveContainer" containerID="c33571d4623196ad2774f4015575df5ca45e881aac105368884d7bc40243b712"
Nov 03 03:14:45.984753 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:45.984730    5005 scope.go:122] "RemoveContainer" containerID="c33571d4623196ad2774f4015575df5ca45e881aac105368884d7bc40243b712"
Nov 03 03:14:45.985056 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:14:45.985035    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c33571d4623196ad2774f4015575df5ca45e881aac105368884d7bc40243b712\": container with ID starting with c33571d4623196ad2774f4015575df5ca45e881aac105368884d7bc40243b712 not found: ID does not exist" containerID="c33571d4623196ad2774f4015575df5ca45e881aac105368884d7bc40243b712"
Nov 03 03:14:45.985104 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:45.985071    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33571d4623196ad2774f4015575df5ca45e881aac105368884d7bc40243b712"} err="failed to get container status \"c33571d4623196ad2774f4015575df5ca45e881aac105368884d7bc40243b712\": rpc error: code = NotFound desc = could not find container \"c33571d4623196ad2774f4015575df5ca45e881aac105368884d7bc40243b712\": container with ID starting with c33571d4623196ad2774f4015575df5ca45e881aac105368884d7bc40243b712 not found: ID does not exist"
Nov 03 03:14:46.012947 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:46.012185    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-kube-api-access-xnlxq\" (UniqueName: \"kubernetes.io/projected/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-kube-api-access-xnlxq\") pod \"81bdbd55-f2ba-4334-8c9f-42d61fea3b33\" (UID: \"81bdbd55-f2ba-4334-8c9f-42d61fea3b33\") "
Nov 03 03:14:46.012947 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:46.012324    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-metrics-certs\" (UniqueName: \"kubernetes.io/secret/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-metrics-certs\") pod \"81bdbd55-f2ba-4334-8c9f-42d61fea3b33\" (UID: \"81bdbd55-f2ba-4334-8c9f-42d61fea3b33\") "
Nov 03 03:14:46.012947 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:46.012390    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-maas-api-tls\") pod \"81bdbd55-f2ba-4334-8c9f-42d61fea3b33\" (UID: \"81bdbd55-f2ba-4334-8c9f-42d61fea3b33\") "
Nov 03 03:14:46.012947 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:46.012451    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-openshift-service-ca\") pod \"81bdbd55-f2ba-4334-8c9f-42d61fea3b33\" (UID: \"81bdbd55-f2ba-4334-8c9f-42d61fea3b33\") "
Nov 03 03:14:46.013351 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:46.013251    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-openshift-service-ca" pod "81bdbd55-f2ba-4334-8c9f-42d61fea3b33" (UID: "81bdbd55-f2ba-4334-8c9f-42d61fea3b33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:14:46.037357 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:46.035763    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-metrics-certs" pod "81bdbd55-f2ba-4334-8c9f-42d61fea3b33" (UID: "81bdbd55-f2ba-4334-8c9f-42d61fea3b33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:14:46.037357 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:46.036564    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-maas-api-tls" pod "81bdbd55-f2ba-4334-8c9f-42d61fea3b33" (UID: "81bdbd55-f2ba-4334-8c9f-42d61fea3b33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:14:46.038141 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:46.038100    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-kube-api-access-xnlxq" pod "81bdbd55-f2ba-4334-8c9f-42d61fea3b33" (UID: "81bdbd55-f2ba-4334-8c9f-42d61fea3b33"). InnerVolumeSpecName "kube-api-access-xnlxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:14:46.113343 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:46.113310    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:14:46.113343 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:46.113339    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:14:46.113343 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:46.113349    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnlxq\" (UniqueName: \"kubernetes.io/projected/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-kube-api-access-xnlxq\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:14:46.113628 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:46.113359    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81bdbd55-f2ba-4334-8c9f-42d61fea3b33-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:14:46.389145 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:46.389102    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"]
Nov 03 03:14:46.392677 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:46.392644    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-521f1f-59c8f5b9df-2rklw"]
Nov 03 03:14:47.108588 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:14:47.108547    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81bdbd55-f2ba-4334-8c9f-42d61fea3b33" path="/var/lib/kubelet/pods/81bdbd55-f2ba-4334-8c9f-42d61fea3b33/volumes"
Nov 03 03:15:00.714067 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:15:00.714025    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:15:07.053500 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:07.053454    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"]
Nov 03 03:15:07.054040 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:07.053752    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n" podUID="a9e03236-c4ee-4370-9b6c-fcb997e3d4b5" containerName="istio-proxy" containerID="cri-o://d8453d044a9c8507bfc59d07cc4502a20acb13b1ae5a1fdbe667e1e62517575d" gracePeriod=30
Nov 03 03:15:08.867900 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:08.867866    5005 patch_prober.go:29] interesting pod/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 03:15:08.868321 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:08.867974    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n" podUID="a9e03236-c4ee-4370-9b6c-fcb997e3d4b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 03:15:09.108196 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:09.108167    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-sxxkw_75f9fa81-69b8-4cfe-acdb-8e68d6e93d02/kuadrant-console-plugin/0.log"
Nov 03 03:15:11.758296 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:11.758269    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-n4klg_e9e336ab-f712-4f2c-b897-fc298e3bfb99/manager/0.log"
Nov 03 03:15:12.314281 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.314249    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 03:15:12.392611 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.392527    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-kube-api-access-gwg8j\" (UniqueName: \"kubernetes.io/projected/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-kube-api-access-gwg8j\") pod \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") "
Nov 03 03:15:12.392611 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.392567    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istiod-ca-cert\") pod \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") "
Nov 03 03:15:12.392611 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.392602    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-token\" (UniqueName: \"kubernetes.io/projected/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-token\") pod \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") "
Nov 03 03:15:12.392910 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.392641    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-podinfo\") pod \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") "
Nov 03 03:15:12.392910 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.392667    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-credential-socket\") pod \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") "
Nov 03 03:15:12.392910 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.392735    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-workload-certs\") pod \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") "
Nov 03 03:15:12.392910 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.392772    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-data\") pod \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") "
Nov 03 03:15:12.392910 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.392850    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-workload-socket\") pod \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") "
Nov 03 03:15:12.393183 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.393051    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-workload-certs" pod "a9e03236-c4ee-4370-9b6c-fcb997e3d4b5" (UID: "a9e03236-c4ee-4370-9b6c-fcb997e3d4b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:15:12.393183 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.393063    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-credential-socket" pod "a9e03236-c4ee-4370-9b6c-fcb997e3d4b5" (UID: "a9e03236-c4ee-4370-9b6c-fcb997e3d4b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:15:12.393183 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.393089    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-workload-socket" pod "a9e03236-c4ee-4370-9b6c-fcb997e3d4b5" (UID: "a9e03236-c4ee-4370-9b6c-fcb997e3d4b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:15:12.393183 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.393154    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-envoy\") pod \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\" (UID: \"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5\") "
Nov 03 03:15:12.393401 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.393307    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-data" pod "a9e03236-c4ee-4370-9b6c-fcb997e3d4b5" (UID: "a9e03236-c4ee-4370-9b6c-fcb997e3d4b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:15:12.393618 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.393591    5005 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-credential-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:15:12.393780 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.393620    5005 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-workload-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:15:12.393780 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.393636    5005 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-data\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:15:12.393780 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.393645    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istiod-ca-cert" pod "a9e03236-c4ee-4370-9b6c-fcb997e3d4b5" (UID: "a9e03236-c4ee-4370-9b6c-fcb997e3d4b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:15:12.393780 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.393650    5005 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-workload-socket\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:15:12.394928 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.394895    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-token" pod "a9e03236-c4ee-4370-9b6c-fcb997e3d4b5" (UID: "a9e03236-c4ee-4370-9b6c-fcb997e3d4b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:15:12.395034 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.394978    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-podinfo" pod "a9e03236-c4ee-4370-9b6c-fcb997e3d4b5" (UID: "a9e03236-c4ee-4370-9b6c-fcb997e3d4b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:15:12.395232 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.395215    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-kube-api-access-gwg8j" pod "a9e03236-c4ee-4370-9b6c-fcb997e3d4b5" (UID: "a9e03236-c4ee-4370-9b6c-fcb997e3d4b5"). InnerVolumeSpecName "kube-api-access-gwg8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:15:12.395334 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.395319    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-envoy" pod "a9e03236-c4ee-4370-9b6c-fcb997e3d4b5" (UID: "a9e03236-c4ee-4370-9b6c-fcb997e3d4b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:15:12.494980 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.494945    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwg8j\" (UniqueName: \"kubernetes.io/projected/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-kube-api-access-gwg8j\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:15:12.494980 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.494975    5005 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istiod-ca-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:15:12.494980 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.494986    5005 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-token\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:15:12.495220 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.494995    5005 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-podinfo\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:15:12.495220 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:12.495004    5005 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5-istio-envoy\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:15:13.043211 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:13.043177    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-594d9845fc-rcqmw_368e9ea7-6100-4196-99af-a84e5901adb9/manager/0.log"
Nov 03 03:15:13.128943 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:13.128903    5005 generic.go:356] "Generic (PLEG): container finished" podID="a9e03236-c4ee-4370-9b6c-fcb997e3d4b5" containerID="d8453d044a9c8507bfc59d07cc4502a20acb13b1ae5a1fdbe667e1e62517575d" exitCode=0
Nov 03 03:15:13.129105 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:13.128962    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"
Nov 03 03:15:13.129105 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:13.128983    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n" event={"ID":"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5","Type":"ContainerDied","Data":"d8453d044a9c8507bfc59d07cc4502a20acb13b1ae5a1fdbe667e1e62517575d"}
Nov 03 03:15:13.129105 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:13.129068    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n" event={"ID":"a9e03236-c4ee-4370-9b6c-fcb997e3d4b5","Type":"ContainerDied","Data":"336e08caa8623cd0b7574b657fa6079e8307b5c4bbabdeb1061a119d4a20a1b2"}
Nov 03 03:15:13.129105 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:13.129098    5005 scope.go:122] "RemoveContainer" containerID="d8453d044a9c8507bfc59d07cc4502a20acb13b1ae5a1fdbe667e1e62517575d"
Nov 03 03:15:13.141156 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:13.141122    5005 scope.go:122] "RemoveContainer" containerID="d8453d044a9c8507bfc59d07cc4502a20acb13b1ae5a1fdbe667e1e62517575d"
Nov 03 03:15:13.141484 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:15:13.141460    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8453d044a9c8507bfc59d07cc4502a20acb13b1ae5a1fdbe667e1e62517575d\": container with ID starting with d8453d044a9c8507bfc59d07cc4502a20acb13b1ae5a1fdbe667e1e62517575d not found: ID does not exist" containerID="d8453d044a9c8507bfc59d07cc4502a20acb13b1ae5a1fdbe667e1e62517575d"
Nov 03 03:15:13.141580 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:13.141492    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8453d044a9c8507bfc59d07cc4502a20acb13b1ae5a1fdbe667e1e62517575d"} err="failed to get container status \"d8453d044a9c8507bfc59d07cc4502a20acb13b1ae5a1fdbe667e1e62517575d\": rpc error: code = NotFound desc = could not find container \"d8453d044a9c8507bfc59d07cc4502a20acb13b1ae5a1fdbe667e1e62517575d\": container with ID starting with d8453d044a9c8507bfc59d07cc4502a20acb13b1ae5a1fdbe667e1e62517575d not found: ID does not exist"
Nov 03 03:15:13.266179 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:13.266145    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-kzmvf_f5a529b5-98e0-433d-ad13-382685e55c5f/server/0.log"
Nov 03 03:15:13.330759 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:13.330660    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"]
Nov 03 03:15:13.377099 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:13.377045    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-521f1f-openshift-default-7c68d87547-jkj2n"]
Nov 03 03:15:13.499414 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:13.499381    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kzgz4_a82e336a-5128-4b44-add6-ea01d6179258/manager/0.log"
Nov 03 03:15:13.754452 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:13.754424    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kflsp_e3a46aa4-c8f9-4fdd-a80a-19f100584149/proxy/0.log"
Nov 03 03:15:15.108450 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:15:15.108411    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9e03236-c4ee-4370-9b6c-fcb997e3d4b5" path="/var/lib/kubelet/pods/a9e03236-c4ee-4370-9b6c-fcb997e3d4b5/volumes"
Nov 03 03:15:30.714192 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:15:30.714150    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:16:00.713422 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:16:00.713331    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:16:13.844855 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:13.844748    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 03:16:13.877859 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:13.877830    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 03:16:16.499358 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:16.499325    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-594d9845fc-rcqmw"]
Nov 03 03:16:16.499816 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:16.499522    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-594d9845fc-rcqmw" podUID="368e9ea7-6100-4196-99af-a84e5901adb9" containerName="manager" containerID="cri-o://3533fe803415e517d136a91cdc597fea0e956cf08c6f3e768dfd9d8c259ac4f1" gracePeriod=30
Nov 03 03:16:16.759954 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:16.759927    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-594d9845fc-rcqmw"
Nov 03 03:16:16.839857 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:16.839823    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/368e9ea7-6100-4196-99af-a84e5901adb9-kube-api-access-25rqw\" (UniqueName: \"kubernetes.io/projected/368e9ea7-6100-4196-99af-a84e5901adb9-kube-api-access-25rqw\") pod \"368e9ea7-6100-4196-99af-a84e5901adb9\" (UID: \"368e9ea7-6100-4196-99af-a84e5901adb9\") "
Nov 03 03:16:16.840078 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:16.839882    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/368e9ea7-6100-4196-99af-a84e5901adb9-webhook-cert\" (UniqueName: \"kubernetes.io/secret/368e9ea7-6100-4196-99af-a84e5901adb9-webhook-cert\") pod \"368e9ea7-6100-4196-99af-a84e5901adb9\" (UID: \"368e9ea7-6100-4196-99af-a84e5901adb9\") "
Nov 03 03:16:16.840078 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:16.839998    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/368e9ea7-6100-4196-99af-a84e5901adb9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/368e9ea7-6100-4196-99af-a84e5901adb9-metrics-certs\") pod \"368e9ea7-6100-4196-99af-a84e5901adb9\" (UID: \"368e9ea7-6100-4196-99af-a84e5901adb9\") "
Nov 03 03:16:16.842010 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:16.841978    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/368e9ea7-6100-4196-99af-a84e5901adb9-metrics-certs" pod "368e9ea7-6100-4196-99af-a84e5901adb9" (UID: "368e9ea7-6100-4196-99af-a84e5901adb9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:16:16.842132 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:16.842011    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/368e9ea7-6100-4196-99af-a84e5901adb9-webhook-cert" pod "368e9ea7-6100-4196-99af-a84e5901adb9" (UID: "368e9ea7-6100-4196-99af-a84e5901adb9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:16:16.842132 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:16.842084    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/368e9ea7-6100-4196-99af-a84e5901adb9-kube-api-access-25rqw" pod "368e9ea7-6100-4196-99af-a84e5901adb9" (UID: "368e9ea7-6100-4196-99af-a84e5901adb9"). InnerVolumeSpecName "kube-api-access-25rqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:16:16.941006 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:16.940963    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/368e9ea7-6100-4196-99af-a84e5901adb9-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:16:16.941006 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:16.940996    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25rqw\" (UniqueName: \"kubernetes.io/projected/368e9ea7-6100-4196-99af-a84e5901adb9-kube-api-access-25rqw\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:16:16.941006 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:16.941006    5005 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/368e9ea7-6100-4196-99af-a84e5901adb9-webhook-cert\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:16:17.484163 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:17.484062    5005 generic.go:356] "Generic (PLEG): container finished" podID="368e9ea7-6100-4196-99af-a84e5901adb9" containerID="3533fe803415e517d136a91cdc597fea0e956cf08c6f3e768dfd9d8c259ac4f1" exitCode=0
Nov 03 03:16:17.484163 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:17.484108    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-594d9845fc-rcqmw"
Nov 03 03:16:17.484163 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:17.484151    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-594d9845fc-rcqmw" event={"ID":"368e9ea7-6100-4196-99af-a84e5901adb9","Type":"ContainerDied","Data":"3533fe803415e517d136a91cdc597fea0e956cf08c6f3e768dfd9d8c259ac4f1"}
Nov 03 03:16:17.484449 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:17.484208    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-594d9845fc-rcqmw" event={"ID":"368e9ea7-6100-4196-99af-a84e5901adb9","Type":"ContainerDied","Data":"2f90e28eec4709257a59c41481515b895fca74f5de7f0e4101d75067b20cfa48"}
Nov 03 03:16:17.484449 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:17.484231    5005 scope.go:122] "RemoveContainer" containerID="3533fe803415e517d136a91cdc597fea0e956cf08c6f3e768dfd9d8c259ac4f1"
Nov 03 03:16:17.495099 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:17.495075    5005 scope.go:122] "RemoveContainer" containerID="3533fe803415e517d136a91cdc597fea0e956cf08c6f3e768dfd9d8c259ac4f1"
Nov 03 03:16:17.495377 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:16:17.495353    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3533fe803415e517d136a91cdc597fea0e956cf08c6f3e768dfd9d8c259ac4f1\": container with ID starting with 3533fe803415e517d136a91cdc597fea0e956cf08c6f3e768dfd9d8c259ac4f1 not found: ID does not exist" containerID="3533fe803415e517d136a91cdc597fea0e956cf08c6f3e768dfd9d8c259ac4f1"
Nov 03 03:16:17.495453 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:17.495385    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3533fe803415e517d136a91cdc597fea0e956cf08c6f3e768dfd9d8c259ac4f1"} err="failed to get container status \"3533fe803415e517d136a91cdc597fea0e956cf08c6f3e768dfd9d8c259ac4f1\": rpc error: code = NotFound desc = could not find container \"3533fe803415e517d136a91cdc597fea0e956cf08c6f3e768dfd9d8c259ac4f1\": container with ID starting with 3533fe803415e517d136a91cdc597fea0e956cf08c6f3e768dfd9d8c259ac4f1 not found: ID does not exist"
Nov 03 03:16:17.637845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:17.637806    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-594d9845fc-rcqmw"]
Nov 03 03:16:17.684199 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:17.684160    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-594d9845fc-rcqmw"]
Nov 03 03:16:19.108871 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:16:19.108837    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="368e9ea7-6100-4196-99af-a84e5901adb9" path="/var/lib/kubelet/pods/368e9ea7-6100-4196-99af-a84e5901adb9/volumes"
Nov 03 03:16:30.713747 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:16:30.713704    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:17:00.714029 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:17:00.713989    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:17:02.366054 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366019    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"]
Nov 03 03:17:02.366451 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366438    5005 cpu_manager.go:414] "containerMap: removing container" podUID="5618cf05-a244-4531-9eca-ee70c484da3e" containerName="maas-api"
Nov 03 03:17:02.366451 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366449    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5618cf05-a244-4531-9eca-ee70c484da3e" containerName="maas-api"
Nov 03 03:17:02.366525 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366462    5005 cpu_manager.go:414] "containerMap: removing container" podUID="1c21f738-1458-4b56-880f-e4ff7fc8123a" containerName="maas-api"
Nov 03 03:17:02.366525 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366467    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c21f738-1458-4b56-880f-e4ff7fc8123a" containerName="maas-api"
Nov 03 03:17:02.366525 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366480    5005 cpu_manager.go:414] "containerMap: removing container" podUID="43baeed2-8514-4f19-90c6-e74f971f32b0" containerName="maas-api"
Nov 03 03:17:02.366525 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366485    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43baeed2-8514-4f19-90c6-e74f971f32b0" containerName="maas-api"
Nov 03 03:17:02.366525 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366493    5005 cpu_manager.go:414] "containerMap: removing container" podUID="008a4c72-1f2f-4fed-ae45-0ab403315d94" containerName="maas-api"
Nov 03 03:17:02.366525 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366498    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="008a4c72-1f2f-4fed-ae45-0ab403315d94" containerName="maas-api"
Nov 03 03:17:02.366525 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366507    5005 cpu_manager.go:414] "containerMap: removing container" podUID="a8b19c9c-11e2-4427-8229-86457df5ebfe" containerName="maas-api"
Nov 03 03:17:02.366525 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366512    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b19c9c-11e2-4427-8229-86457df5ebfe" containerName="maas-api"
Nov 03 03:17:02.366525 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366521    5005 cpu_manager.go:414] "containerMap: removing container" podUID="700d764b-0150-4711-aa7c-718170d233d5" containerName="istio-proxy"
Nov 03 03:17:02.366525 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366528    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700d764b-0150-4711-aa7c-718170d233d5" containerName="istio-proxy"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366537    5005 cpu_manager.go:414] "containerMap: removing container" podUID="368e9ea7-6100-4196-99af-a84e5901adb9" containerName="manager"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366542    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="368e9ea7-6100-4196-99af-a84e5901adb9" containerName="manager"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366548    5005 cpu_manager.go:414] "containerMap: removing container" podUID="f2b37f9f-b995-403b-8ba8-e27273057f9f" containerName="maas-api"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366553    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b37f9f-b995-403b-8ba8-e27273057f9f" containerName="maas-api"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366560    5005 cpu_manager.go:414] "containerMap: removing container" podUID="c58a3573-fd82-4f92-829b-475591248445" containerName="maas-api"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366564    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c58a3573-fd82-4f92-829b-475591248445" containerName="maas-api"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366571    5005 cpu_manager.go:414] "containerMap: removing container" podUID="55f0246d-7f9d-4581-bd2b-c98e20e97676" containerName="istio-proxy"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366575    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f0246d-7f9d-4581-bd2b-c98e20e97676" containerName="istio-proxy"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366581    5005 cpu_manager.go:414] "containerMap: removing container" podUID="f8f6fafc-574b-47e5-b63a-0eed748772ee" containerName="payload-processing"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366585    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f6fafc-574b-47e5-b63a-0eed748772ee" containerName="payload-processing"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366593    5005 cpu_manager.go:414] "containerMap: removing container" podUID="7f39ea17-c4d7-4de2-bb69-c7d3012c5f77" containerName="main"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366597    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f39ea17-c4d7-4de2-bb69-c7d3012c5f77" containerName="main"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366602    5005 cpu_manager.go:414] "containerMap: removing container" podUID="d2e9fe34-52c0-431a-8088-2783488fa3f0" containerName="maas-api"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366606    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2e9fe34-52c0-431a-8088-2783488fa3f0" containerName="maas-api"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366616    5005 cpu_manager.go:414] "containerMap: removing container" podUID="394a32fc-4122-44a7-aef5-937f6f691da2" containerName="payload-pre-processing"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366621    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="394a32fc-4122-44a7-aef5-937f6f691da2" containerName="payload-pre-processing"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366628    5005 cpu_manager.go:414] "containerMap: removing container" podUID="6a37751f-fd21-4160-a44f-e97a796f9198" containerName="istio-proxy"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366632    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a37751f-fd21-4160-a44f-e97a796f9198" containerName="istio-proxy"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366639    5005 cpu_manager.go:414] "containerMap: removing container" podUID="b80d8879-30b7-477c-8b8e-5c806d813e54" containerName="istio-proxy"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366643    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b80d8879-30b7-477c-8b8e-5c806d813e54" containerName="istio-proxy"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366656    5005 cpu_manager.go:414] "containerMap: removing container" podUID="81bdbd55-f2ba-4334-8c9f-42d61fea3b33" containerName="maas-api"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366660    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81bdbd55-f2ba-4334-8c9f-42d61fea3b33" containerName="maas-api"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366665    5005 cpu_manager.go:414] "containerMap: removing container" podUID="41a24e80-e0db-4bee-a1e8-eec710e6dd79" containerName="maas-api"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366669    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a24e80-e0db-4bee-a1e8-eec710e6dd79" containerName="maas-api"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366676    5005 cpu_manager.go:414] "containerMap: removing container" podUID="eded8d0f-e8e1-4410-baa9-e2982338f3e3" containerName="maas-api"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366680    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eded8d0f-e8e1-4410-baa9-e2982338f3e3" containerName="maas-api"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366686    5005 cpu_manager.go:414] "containerMap: removing container" podUID="22dc475f-cfa4-435c-b021-ccdd2f2eec91" containerName="maas-api"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366690    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22dc475f-cfa4-435c-b021-ccdd2f2eec91" containerName="maas-api"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366711    5005 cpu_manager.go:414] "containerMap: removing container" podUID="532d5d68-bc86-491b-978b-97f9cf54580a" containerName="maas-api"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366716    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="532d5d68-bc86-491b-978b-97f9cf54580a" containerName="maas-api"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366722    5005 cpu_manager.go:414] "containerMap: removing container" podUID="f0019a0f-0716-4bd2-be7b-f96ff62bf7d1" containerName="istio-proxy"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366726    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0019a0f-0716-4bd2-be7b-f96ff62bf7d1" containerName="istio-proxy"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366734    5005 cpu_manager.go:414] "containerMap: removing container" podUID="b1f526d9-2fd8-4c8d-9175-732afdbf3570" containerName="main"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366738    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1f526d9-2fd8-4c8d-9175-732afdbf3570" containerName="main"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366747    5005 cpu_manager.go:414] "containerMap: removing container" podUID="fbdeec42-7115-4361-ac6f-1874ffded23c" containerName="maas-api"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366751    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbdeec42-7115-4361-ac6f-1874ffded23c" containerName="maas-api"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366758    5005 cpu_manager.go:414] "containerMap: removing container" podUID="a0b69c55-a8e5-4fb9-8e61-e8533ee739a2" containerName="maas-api"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366762    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b69c55-a8e5-4fb9-8e61-e8533ee739a2" containerName="maas-api"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366771    5005 cpu_manager.go:414] "containerMap: removing container" podUID="fd71fd74-9581-4c83-8954-55462e1d4438" containerName="main"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366775    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd71fd74-9581-4c83-8954-55462e1d4438" containerName="main"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366780    5005 cpu_manager.go:414] "containerMap: removing container" podUID="a9e03236-c4ee-4370-9b6c-fcb997e3d4b5" containerName="istio-proxy"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366785    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e03236-c4ee-4370-9b6c-fcb997e3d4b5" containerName="istio-proxy"
Nov 03 03:17:02.366845 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366862    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0019a0f-0716-4bd2-be7b-f96ff62bf7d1" containerName="istio-proxy"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366871    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1f526d9-2fd8-4c8d-9175-732afdbf3570" containerName="main"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366878    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbdeec42-7115-4361-ac6f-1874ffded23c" containerName="maas-api"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366884    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b69c55-a8e5-4fb9-8e61-e8533ee739a2" containerName="maas-api"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366890    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd71fd74-9581-4c83-8954-55462e1d4438" containerName="main"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366895    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e03236-c4ee-4370-9b6c-fcb997e3d4b5" containerName="istio-proxy"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366901    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="5618cf05-a244-4531-9eca-ee70c484da3e" containerName="maas-api"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366909    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c21f738-1458-4b56-880f-e4ff7fc8123a" containerName="maas-api"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366921    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="43baeed2-8514-4f19-90c6-e74f971f32b0" containerName="maas-api"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366929    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="008a4c72-1f2f-4fed-ae45-0ab403315d94" containerName="maas-api"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366937    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b19c9c-11e2-4427-8229-86457df5ebfe" containerName="maas-api"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366945    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="700d764b-0150-4711-aa7c-718170d233d5" containerName="istio-proxy"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366970    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="368e9ea7-6100-4196-99af-a84e5901adb9" containerName="manager"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366977    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b37f9f-b995-403b-8ba8-e27273057f9f" containerName="maas-api"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366985    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="c58a3573-fd82-4f92-829b-475591248445" containerName="maas-api"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366993    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f0246d-7f9d-4581-bd2b-c98e20e97676" containerName="istio-proxy"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.366999    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f6fafc-574b-47e5-b63a-0eed748772ee" containerName="payload-processing"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.367004    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f39ea17-c4d7-4de2-bb69-c7d3012c5f77" containerName="main"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.367010    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2e9fe34-52c0-431a-8088-2783488fa3f0" containerName="maas-api"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.367016    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="394a32fc-4122-44a7-aef5-937f6f691da2" containerName="payload-pre-processing"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.367022    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a37751f-fd21-4160-a44f-e97a796f9198" containerName="istio-proxy"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.367027    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="b80d8879-30b7-477c-8b8e-5c806d813e54" containerName="istio-proxy"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.367034    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="81bdbd55-f2ba-4334-8c9f-42d61fea3b33" containerName="maas-api"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.367041    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a24e80-e0db-4bee-a1e8-eec710e6dd79" containerName="maas-api"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.367046    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="eded8d0f-e8e1-4410-baa9-e2982338f3e3" containerName="maas-api"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.367051    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="22dc475f-cfa4-435c-b021-ccdd2f2eec91" containerName="maas-api"
Nov 03 03:17:02.368462 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.367060    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="532d5d68-bc86-491b-978b-97f9cf54580a" containerName="maas-api"
Nov 03 03:17:02.370290 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.370275    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.506038 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.506003    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjcjv\" (UniqueName: \"kubernetes.io/projected/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-kube-api-access-vjcjv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.506222 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.506043    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.506222 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.506066    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.506222 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.506160    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.506349 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.506250    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.506349 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.506343    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.506418 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.506367    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.506418 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.506388    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.506588 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.506413    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.564741 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.564686    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"]
Nov 03 03:17:02.607252 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.607215    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.607252 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.607256    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.607491 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.607277    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.607491 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.607297    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.607491 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.607367    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjcjv\" (UniqueName: \"kubernetes.io/projected/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-kube-api-access-vjcjv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.607491 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.607395    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.607491 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.607423    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.607491 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.607442    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.607491 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.607481    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.607906 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.607746    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.607906 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.607834    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.607906 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.607873    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.608067 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.607987    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.608125 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.608101    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.609835 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.609811    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.610013 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.609993    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.661473 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.661398    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.661763 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.661743    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjcjv\" (UniqueName: \"kubernetes.io/projected/239e8691-a4ce-4e72-ac2d-5dbc72b182d7-kube-api-access-vjcjv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-vpjhq\" (UID: \"239e8691-a4ce-4e72-ac2d-5dbc72b182d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:02.685543 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:02.685513    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:03.109792 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:03.109746    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"]
Nov 03 03:17:03.114433 ip-10-0-139-26 kubenswrapper[5005]: W1103 03:17:03.114405    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239e8691_a4ce_4e72_ac2d_5dbc72b182d7.slice/crio-0793729a69e5d8e7ca24972300dd4cad040ec3f014023c03dcfee3b7ca0d3a3f WatchSource:0}: Error finding container 0793729a69e5d8e7ca24972300dd4cad040ec3f014023c03dcfee3b7ca0d3a3f: Status 404 returned error can't find the container with id 0793729a69e5d8e7ca24972300dd4cad040ec3f014023c03dcfee3b7ca0d3a3f
Nov 03 03:17:03.116443 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:03.116408    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 03:17:03.116541 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:03.116476    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 03:17:03.116541 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:03.116507    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 03:17:03.730461 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:03.730419    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq" event={"ID":"239e8691-a4ce-4e72-ac2d-5dbc72b182d7","Type":"ContainerStarted","Data":"1b36cf036e7e1b8b3cbce21f6d68564461b5f010ef071e7fb6962969e5ed0a7f"}
Nov 03 03:17:03.730910 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:03.730490    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq" event={"ID":"239e8691-a4ce-4e72-ac2d-5dbc72b182d7","Type":"ContainerStarted","Data":"0793729a69e5d8e7ca24972300dd4cad040ec3f014023c03dcfee3b7ca0d3a3f"}
Nov 03 03:17:03.841734 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:03.841658    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq" podStartSLOduration=1.8416432820000002 podStartE2EDuration="1.841643282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:17:03.840266449 +0000 UTC m=+5150.854507813" watchObservedRunningTime="2026-11-03 03:17:03.841643282 +0000 UTC m=+5150.855884645"
Nov 03 03:17:04.686184 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:04.686142    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:04.688072 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:04.688044    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-vpjhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 03 03:17:04.688207 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:04.688107    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq" podUID="239e8691-a4ce-4e72-ac2d-5dbc72b182d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 03 03:17:05.686103 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:05.686061    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-vpjhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 03 03:17:05.686521 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:05.686119    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq" podUID="239e8691-a4ce-4e72-ac2d-5dbc72b182d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 03 03:17:06.686295 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:06.686260    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-vpjhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 03 03:17:06.686748 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:06.686324    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq" podUID="239e8691-a4ce-4e72-ac2d-5dbc72b182d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 03 03:17:07.686922 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:07.686874    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-vpjhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 03 03:17:07.687392 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:07.686939    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq" podUID="239e8691-a4ce-4e72-ac2d-5dbc72b182d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 03 03:17:08.686199 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:08.686153    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-vpjhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 03 03:17:08.686388 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:08.686219    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq" podUID="239e8691-a4ce-4e72-ac2d-5dbc72b182d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 03 03:17:09.694817 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:09.694767    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-vpjhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 03 03:17:09.695356 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:09.694853    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq" podUID="239e8691-a4ce-4e72-ac2d-5dbc72b182d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 03 03:17:11.487979 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:11.487941    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:11.488440 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:11.488376    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:11.489136 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:11.489114    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq"
Nov 03 03:17:30.715400 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:17:30.715315    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:17:47.706940 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.706895    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"]
Nov 03 03:17:47.711144 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.711118    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.774440 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.774398    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32593158-9c57-4ecc-8a47-a7ca894ed169-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.774440 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.774441    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32593158-9c57-4ecc-8a47-a7ca894ed169-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.774657 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.774543    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2gfd\" (UniqueName: \"kubernetes.io/projected/32593158-9c57-4ecc-8a47-a7ca894ed169-kube-api-access-q2gfd\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.774657 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.774573    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32593158-9c57-4ecc-8a47-a7ca894ed169-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.774657 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.774592    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32593158-9c57-4ecc-8a47-a7ca894ed169-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.774800 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.774687    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32593158-9c57-4ecc-8a47-a7ca894ed169-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.774800 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.774750    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32593158-9c57-4ecc-8a47-a7ca894ed169-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.774872 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.774838    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32593158-9c57-4ecc-8a47-a7ca894ed169-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.774872 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.774856    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32593158-9c57-4ecc-8a47-a7ca894ed169-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.875640 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.875599    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2gfd\" (UniqueName: \"kubernetes.io/projected/32593158-9c57-4ecc-8a47-a7ca894ed169-kube-api-access-q2gfd\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.875915 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.875650    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32593158-9c57-4ecc-8a47-a7ca894ed169-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.875915 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.875759    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32593158-9c57-4ecc-8a47-a7ca894ed169-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.875915 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.875824    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32593158-9c57-4ecc-8a47-a7ca894ed169-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.875915 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.875877    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32593158-9c57-4ecc-8a47-a7ca894ed169-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.876158 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.875982    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32593158-9c57-4ecc-8a47-a7ca894ed169-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.876158 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.876019    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32593158-9c57-4ecc-8a47-a7ca894ed169-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.876158 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.876101    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32593158-9c57-4ecc-8a47-a7ca894ed169-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.876158 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.876130    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32593158-9c57-4ecc-8a47-a7ca894ed169-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.876388 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.876361    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32593158-9c57-4ecc-8a47-a7ca894ed169-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.876476 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.876450    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32593158-9c57-4ecc-8a47-a7ca894ed169-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.876637 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.876468    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32593158-9c57-4ecc-8a47-a7ca894ed169-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.876637 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.876578    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32593158-9c57-4ecc-8a47-a7ca894ed169-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.876819 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.876795    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32593158-9c57-4ecc-8a47-a7ca894ed169-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.878112 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.878093    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32593158-9c57-4ecc-8a47-a7ca894ed169-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.878357 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.878340    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32593158-9c57-4ecc-8a47-a7ca894ed169-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.898356 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.898326    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"]
Nov 03 03:17:47.930312 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.930278    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32593158-9c57-4ecc-8a47-a7ca894ed169-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:47.930604 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:47.930578    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2gfd\" (UniqueName: \"kubernetes.io/projected/32593158-9c57-4ecc-8a47-a7ca894ed169-kube-api-access-q2gfd\") pod \"maas-default-gateway-openshift-default-5bc884dd87-f72x6\" (UID: \"32593158-9c57-4ecc-8a47-a7ca894ed169\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:48.024899 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:48.024863    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:48.454923 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:48.454831    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"]
Nov 03 03:17:48.461777 ip-10-0-139-26 kubenswrapper[5005]: W1103 03:17:48.461742    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32593158_9c57_4ecc_8a47_a7ca894ed169.slice/crio-414fee4be333e2d1d487fb2a7a869a5f64a12e1f31fd4ff6d1fa78292583375b WatchSource:0}: Error finding container 414fee4be333e2d1d487fb2a7a869a5f64a12e1f31fd4ff6d1fa78292583375b: Status 404 returned error can't find the container with id 414fee4be333e2d1d487fb2a7a869a5f64a12e1f31fd4ff6d1fa78292583375b
Nov 03 03:17:48.463951 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:48.463913    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 03:17:48.464079 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:48.463981    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 03:17:48.464079 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:48.464008    5005 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 03:17:48.995413 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:48.995364    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6" event={"ID":"32593158-9c57-4ecc-8a47-a7ca894ed169","Type":"ContainerStarted","Data":"6024abfd8337c415176a22579d05e27253c64c15dbf2caea1a41e9783299ed22"}
Nov 03 03:17:48.995958 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:48.995445    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6" event={"ID":"32593158-9c57-4ecc-8a47-a7ca894ed169","Type":"ContainerStarted","Data":"414fee4be333e2d1d487fb2a7a869a5f64a12e1f31fd4ff6d1fa78292583375b"}
Nov 03 03:17:49.031717 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:49.031115    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:49.034753 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:49.033899    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f72x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 03:17:49.034753 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:49.033961    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6" podUID="32593158-9c57-4ecc-8a47-a7ca894ed169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 03:17:49.191733 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:49.191661    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6" podStartSLOduration=2.191646596 podStartE2EDuration="2.191646596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:17:49.189848715 +0000 UTC m=+5196.204090082" watchObservedRunningTime="2026-11-03 03:17:49.191646596 +0000 UTC m=+5196.205887962"
Nov 03 03:17:50.030238 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:50.030195    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f72x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 03:17:50.030948 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:50.030900    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6" podUID="32593158-9c57-4ecc-8a47-a7ca894ed169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 03:17:51.025509 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:51.025462    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f72x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 03:17:51.025676 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:51.025521    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6" podUID="32593158-9c57-4ecc-8a47-a7ca894ed169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 03:17:52.025911 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:52.025869    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f72x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 03:17:52.026362 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:52.025944    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6" podUID="32593158-9c57-4ecc-8a47-a7ca894ed169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 03:17:53.026283 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:53.026240    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f72x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 03:17:53.026916 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:53.026325    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6" podUID="32593158-9c57-4ecc-8a47-a7ca894ed169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 03:17:54.026228 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:54.026186    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f72x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 03:17:54.026446 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:54.026243    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6" podUID="32593158-9c57-4ecc-8a47-a7ca894ed169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 03:17:55.025956 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:55.025917    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f72x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 03:17:55.026181 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:55.025980    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6" podUID="32593158-9c57-4ecc-8a47-a7ca894ed169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 03:17:56.026140 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:56.026101    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f72x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 03:17:56.026593 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:56.026170    5005 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6" podUID="32593158-9c57-4ecc-8a47-a7ca894ed169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 03:17:57.029577 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:57.029538    5005 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:57.030030 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:57.029901    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:17:57.030475 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:17:57.030453    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6"
Nov 03 03:18:00.713350 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:18:00.713311    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:18:30.713906 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:18:30.713865    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:19:00.713851 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:19:00.713766    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:19:30.714566 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:19:30.714520    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:19:42.530864 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:19:42.530815    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-vpjhq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:19:42.531530 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:19:42.531492    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq" podUID="239e8691-a4ce-4e72-ac2d-5dbc72b182d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:20:00.714288 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:20:00.714245    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:20:13.598733 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:13.597875    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-pd8pt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:20:13.598733 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:13.597964    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt" podUID="ee540784-ffc0-40e2-be04-d8c13a2521b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:20:28.604721 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:28.602836    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-pd8pt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:20:28.604721 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:28.602901    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt" podUID="ee540784-ffc0-40e2-be04-d8c13a2521b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:20:30.714328 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:20:30.714287    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:20:34.930729 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:34.930673    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"]
Nov 03 03:20:34.934641 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:34.934621    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"
Nov 03 03:20:34.986342 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:34.986307    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-2232ad-serving-cert\""
Nov 03 03:20:35.035634 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:35.035600    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"]
Nov 03 03:20:35.046412 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:35.046381    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcfjv\" (UniqueName: \"kubernetes.io/projected/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-kube-api-access-hcfjv\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft\" (UID: \"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"
Nov 03 03:20:35.046412 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:35.046415    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft\" (UID: \"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"
Nov 03 03:20:35.046653 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:35.046450    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft\" (UID: \"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"
Nov 03 03:20:35.046738 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:35.046680    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft\" (UID: \"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"
Nov 03 03:20:35.147924 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:35.147880    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcfjv\" (UniqueName: \"kubernetes.io/projected/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-kube-api-access-hcfjv\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft\" (UID: \"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"
Nov 03 03:20:35.147924 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:35.147925    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft\" (UID: \"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"
Nov 03 03:20:35.148180 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:35.147984    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft\" (UID: \"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"
Nov 03 03:20:35.148180 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:35.148086    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft\" (UID: \"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"
Nov 03 03:20:35.148180 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:20:35.148085    5005 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-serving-cert: secret "maas-api-e2e-worker-single-main-2232ad-serving-cert" not found
Nov 03 03:20:35.148368 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:20:35.148263    5005 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-maas-api-tls podName:2bc1a618-4f5b-4c2a-b731-ef038f51eaa6 nodeName:}" failed. No retries permitted until 2026-11-03 03:20:35.648238291 +0000 UTC m=+5362.662479643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-maas-api-tls") pod "maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft" (UID: "2bc1a618-4f5b-4c2a-b731-ef038f51eaa6") : secret "maas-api-e2e-worker-single-main-2232ad-serving-cert" not found
Nov 03 03:20:35.148742 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:35.148689    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft\" (UID: \"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"
Nov 03 03:20:35.150557 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:35.150532    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft\" (UID: \"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"
Nov 03 03:20:35.203661 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:35.203587    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcfjv\" (UniqueName: \"kubernetes.io/projected/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-kube-api-access-hcfjv\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft\" (UID: \"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"
Nov 03 03:20:35.653722 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:35.653655    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft\" (UID: \"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"
Nov 03 03:20:35.656110 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:35.656082    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft\" (UID: \"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"
Nov 03 03:20:35.849109 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:35.849072    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"
Nov 03 03:20:36.228876 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:36.228789    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"]
Nov 03 03:20:36.234971 ip-10-0-139-26 kubenswrapper[5005]: W1103 03:20:36.234938    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc1a618_4f5b_4c2a_b731_ef038f51eaa6.slice/crio-ef1a25858d626d5a4ab499198dce39bec013304e50d8b3bf4bf2157931e2d802 WatchSource:0}: Error finding container ef1a25858d626d5a4ab499198dce39bec013304e50d8b3bf4bf2157931e2d802: Status 404 returned error can't find the container with id ef1a25858d626d5a4ab499198dce39bec013304e50d8b3bf4bf2157931e2d802
Nov 03 03:20:36.236566 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:36.236548    5005 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 03:20:37.121210 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:37.121169    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft" event={"ID":"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6","Type":"ContainerStarted","Data":"ef1a25858d626d5a4ab499198dce39bec013304e50d8b3bf4bf2157931e2d802"}
Nov 03 03:20:38.128411 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:38.128358    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft" event={"ID":"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6","Type":"ContainerStarted","Data":"6889936d375a2d9649e746e90aed8f9e6e72c32a118cbda77bccc67299a4f743"}
Nov 03 03:20:38.128977 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:38.128423    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"
Nov 03 03:20:38.237900 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:38.237840    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft" podStartSLOduration=3.267620448 podStartE2EDuration="4.237824914s" totalImagesPullingTime="970.204466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:20:38.236734494 +0000 UTC m=+5365.250975861" watchObservedRunningTime="2026-11-03 03:20:38.237824914 +0000 UTC m=+5365.252066279"
Nov 03 03:20:44.140069 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:20:44.140036    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"
Nov 03 03:21:00.713354 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:21:00.713312    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:21:13.882516 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:21:13.882402    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 03:21:13.915225 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:21:13.915197    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 03:21:30.713491 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:21:30.713449    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:22:00.714069 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:22:00.713986    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:22:01.401648 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:01.401607    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"]
Nov 03 03:22:01.401878 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:01.401847    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft" podUID="2bc1a618-4f5b-4c2a-b731-ef038f51eaa6" containerName="maas-api" containerID="cri-o://6889936d375a2d9649e746e90aed8f9e6e72c32a118cbda77bccc67299a4f743" gracePeriod=30
Nov 03 03:22:02.608764 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:02.608732    5005 generic.go:356] "Generic (PLEG): container finished" podID="2bc1a618-4f5b-4c2a-b731-ef038f51eaa6" containerID="6889936d375a2d9649e746e90aed8f9e6e72c32a118cbda77bccc67299a4f743" exitCode=0
Nov 03 03:22:02.609185 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:02.608811    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft" event={"ID":"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6","Type":"ContainerDied","Data":"6889936d375a2d9649e746e90aed8f9e6e72c32a118cbda77bccc67299a4f743"}
Nov 03 03:22:02.656314 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:02.656284    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"
Nov 03 03:22:02.716373 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:02.716337    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-maas-api-tls\") pod \"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6\" (UID: \"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6\") "
Nov 03 03:22:02.716559 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:02.716405    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-kube-api-access-hcfjv\" (UniqueName: \"kubernetes.io/projected/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-kube-api-access-hcfjv\") pod \"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6\" (UID: \"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6\") "
Nov 03 03:22:02.716559 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:02.716481    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-openshift-service-ca\") pod \"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6\" (UID: \"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6\") "
Nov 03 03:22:02.716559 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:02.716523    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-metrics-certs\") pod \"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6\" (UID: \"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6\") "
Nov 03 03:22:02.716869 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:02.716844    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-openshift-service-ca" pod "2bc1a618-4f5b-4c2a-b731-ef038f51eaa6" (UID: "2bc1a618-4f5b-4c2a-b731-ef038f51eaa6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:22:02.716960 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:02.716941    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:22:02.718618 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:02.718588    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-maas-api-tls" pod "2bc1a618-4f5b-4c2a-b731-ef038f51eaa6" (UID: "2bc1a618-4f5b-4c2a-b731-ef038f51eaa6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:22:02.718737 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:02.718615    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-metrics-certs" pod "2bc1a618-4f5b-4c2a-b731-ef038f51eaa6" (UID: "2bc1a618-4f5b-4c2a-b731-ef038f51eaa6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:22:02.718737 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:02.718656    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-kube-api-access-hcfjv" pod "2bc1a618-4f5b-4c2a-b731-ef038f51eaa6" (UID: "2bc1a618-4f5b-4c2a-b731-ef038f51eaa6"). InnerVolumeSpecName "kube-api-access-hcfjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:22:02.818396 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:02.818352    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:22:02.818396 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:02.818389    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcfjv\" (UniqueName: \"kubernetes.io/projected/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-kube-api-access-hcfjv\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:22:02.818396 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:02.818399    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:22:03.615884 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:03.615853    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"
Nov 03 03:22:03.616330 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:03.615861    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft" event={"ID":"2bc1a618-4f5b-4c2a-b731-ef038f51eaa6","Type":"ContainerDied","Data":"ef1a25858d626d5a4ab499198dce39bec013304e50d8b3bf4bf2157931e2d802"}
Nov 03 03:22:03.616330 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:03.615979    5005 scope.go:122] "RemoveContainer" containerID="6889936d375a2d9649e746e90aed8f9e6e72c32a118cbda77bccc67299a4f743"
Nov 03 03:22:03.771130 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:03.771062    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"]
Nov 03 03:22:03.817514 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:03.817479    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2232ad-868868b589-2j5ft"]
Nov 03 03:22:05.108347 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:22:05.108303    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bc1a618-4f5b-4c2a-b731-ef038f51eaa6" path="/var/lib/kubelet/pods/2bc1a618-4f5b-4c2a-b731-ef038f51eaa6/volumes"
Nov 03 03:22:30.713679 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:22:30.713630    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:23:00.714155 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:23:00.714115    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:23:30.714113 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:23:30.714028    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:24:00.712921 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:24:00.712879    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:24:30.782278 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:24:30.782197    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:25:00.715427 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:25:00.715341    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:25:30.714356 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:25:30.714305    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:25:42.780313 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:25:42.780267    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-dn4d4"]
Nov 03 03:25:47.781442 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:25:47.781391    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-dn4d4" podUID="30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2" containerName="payload-processing" containerID="cri-o://77ddd29f0624b7eca34373df58dfdf8a51b57bd720e8ba378ed57790e046aee8" gracePeriod=25
Nov 03 03:25:48.035995 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:25:48.035932    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-dn4d4"
Nov 03 03:25:48.080228 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:25:48.080192    5005 generic.go:356] "Generic (PLEG): container finished" podID="30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2" containerID="77ddd29f0624b7eca34373df58dfdf8a51b57bd720e8ba378ed57790e046aee8" exitCode=0
Nov 03 03:25:48.080431 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:25:48.080238    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-dn4d4"
Nov 03 03:25:48.080431 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:25:48.080278    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-dn4d4" event={"ID":"30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2","Type":"ContainerDied","Data":"77ddd29f0624b7eca34373df58dfdf8a51b57bd720e8ba378ed57790e046aee8"}
Nov 03 03:25:48.080431 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:25:48.080347    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-dn4d4" event={"ID":"30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2","Type":"ContainerDied","Data":"a2b74b61d11fc22d158e0340cfaca7313a4b756e388cfb08db56372c6058d1dc"}
Nov 03 03:25:48.080431 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:25:48.080378    5005 scope.go:122] "RemoveContainer" containerID="77ddd29f0624b7eca34373df58dfdf8a51b57bd720e8ba378ed57790e046aee8"
Nov 03 03:25:48.089767 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:25:48.089747    5005 scope.go:122] "RemoveContainer" containerID="77ddd29f0624b7eca34373df58dfdf8a51b57bd720e8ba378ed57790e046aee8"
Nov 03 03:25:48.090077 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:25:48.090056    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ddd29f0624b7eca34373df58dfdf8a51b57bd720e8ba378ed57790e046aee8\": container with ID starting with 77ddd29f0624b7eca34373df58dfdf8a51b57bd720e8ba378ed57790e046aee8 not found: ID does not exist" containerID="77ddd29f0624b7eca34373df58dfdf8a51b57bd720e8ba378ed57790e046aee8"
Nov 03 03:25:48.090123 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:25:48.090091    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ddd29f0624b7eca34373df58dfdf8a51b57bd720e8ba378ed57790e046aee8"} err="failed to get container status \"77ddd29f0624b7eca34373df58dfdf8a51b57bd720e8ba378ed57790e046aee8\": rpc error: code = NotFound desc = could not find container \"77ddd29f0624b7eca34373df58dfdf8a51b57bd720e8ba378ed57790e046aee8\": container with ID starting with 77ddd29f0624b7eca34373df58dfdf8a51b57bd720e8ba378ed57790e046aee8 not found: ID does not exist"
Nov 03 03:25:48.090322 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:25:48.090308    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2-plugins-config-volume\") pod \"30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2\" (UID: \"30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2\") "
Nov 03 03:25:48.090426 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:25:48.090412    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2-kube-api-access-268vm\" (UniqueName: \"kubernetes.io/projected/30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2-kube-api-access-268vm\") pod \"30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2\" (UID: \"30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2\") "
Nov 03 03:25:48.090765 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:25:48.090736    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2-plugins-config-volume" pod "30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2" (UID: "30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:25:48.092399 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:25:48.092378    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2-kube-api-access-268vm" pod "30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2" (UID: "30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2"). InnerVolumeSpecName "kube-api-access-268vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:25:48.191312 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:25:48.191281    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-268vm\" (UniqueName: \"kubernetes.io/projected/30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2-kube-api-access-268vm\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:25:48.191312 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:25:48.191309    5005 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2-plugins-config-volume\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:25:48.493938 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:25:48.493901    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-dn4d4"]
Nov 03 03:25:48.499815 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:25:48.499786    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-dn4d4"]
Nov 03 03:25:49.108353 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:25:49.108319    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2" path="/var/lib/kubelet/pods/30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2/volumes"
Nov 03 03:26:00.714491 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:26:00.714449    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:26:13.928131 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:26:13.928006    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 03:26:13.953869 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:26:13.953842    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 03:26:30.712965 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:26:30.712881    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:27:00.712754 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:27:00.712712    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:27:30.714345 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:27:30.714303    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:28:00.714908 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:28:00.714864    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:28:30.714713 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:28:30.714657    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:29:00.713465 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:29:00.713421    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:29:30.713666 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:29:30.713580    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:30:00.720811 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:30:00.720769    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:30:30.713249 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:30:30.713210    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:30:58.609720 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:30:58.606821    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-pd8pt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:30:58.609720 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:30:58.606902    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt" podUID="ee540784-ffc0-40e2-be04-d8c13a2521b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:31:00.713170 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:31:00.713126    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:31:13.983383 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:31:13.983265    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 03:31:13.999826 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:31:13.999800    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 03:31:30.714606 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:31:30.714565    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:31:43.075905 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:31:43.075855    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-f72x6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:31:43.076475 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:31:43.075932    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-f72x6" podUID="32593158-9c57-4ecc-8a47-a7ca894ed169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:31:43.594861 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:31:43.594820    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-pd8pt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:31:43.595059 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:31:43.594882    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt" podUID="ee540784-ffc0-40e2-be04-d8c13a2521b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:32:00.713019 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:32:00.712980    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:32:30.714517 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:32:30.714435    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:33:00.713899 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:33:00.713859    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:33:30.713907 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:33:30.713866    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:34:00.714362 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:34:00.714322    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:34:30.714032 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:34:30.713992    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:35:00.714152 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:35:00.714106    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:35:30.713835 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:35:30.713757    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:36:00.713688 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:36:00.713642    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:36:14.021713 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:36:14.021668    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 03:36:14.040021 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:36:14.039994    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 03:36:30.713369 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:36:30.713321    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:37:00.713177 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:37:00.713138    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:37:30.783546 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:37:30.783499    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:37:54.523544 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:54.523498    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfdcdb597-ngddf"]
Nov 03 03:37:54.524245 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:54.524222    5005 cpu_manager.go:414] "containerMap: removing container" podUID="30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2" containerName="payload-processing"
Nov 03 03:37:54.524352 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:54.524250    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2" containerName="payload-processing"
Nov 03 03:37:54.524352 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:54.524274    5005 cpu_manager.go:414] "containerMap: removing container" podUID="2bc1a618-4f5b-4c2a-b731-ef038f51eaa6" containerName="maas-api"
Nov 03 03:37:54.524352 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:54.524283    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bc1a618-4f5b-4c2a-b731-ef038f51eaa6" containerName="maas-api"
Nov 03 03:37:54.524525 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:54.524474    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="30a8658b-0d9d-46cc-a3f5-16fc6fdfd5e2" containerName="payload-processing"
Nov 03 03:37:54.524525 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:54.524489    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bc1a618-4f5b-4c2a-b731-ef038f51eaa6" containerName="maas-api"
Nov 03 03:37:54.528297 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:54.528269    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-ngddf"
Nov 03 03:37:54.614897 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:54.614861    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e23c501-f20c-4d5b-8946-410c65498dfb-metrics-certs\") pod \"maas-api-5bfdcdb597-ngddf\" (UID: \"3e23c501-f20c-4d5b-8946-410c65498dfb\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-ngddf"
Nov 03 03:37:54.615122 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:54.615005    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e23c501-f20c-4d5b-8946-410c65498dfb-maas-api-tls\") pod \"maas-api-5bfdcdb597-ngddf\" (UID: \"3e23c501-f20c-4d5b-8946-410c65498dfb\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-ngddf"
Nov 03 03:37:54.615221 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:54.615193    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82psw\" (UniqueName: \"kubernetes.io/projected/3e23c501-f20c-4d5b-8946-410c65498dfb-kube-api-access-82psw\") pod \"maas-api-5bfdcdb597-ngddf\" (UID: \"3e23c501-f20c-4d5b-8946-410c65498dfb\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-ngddf"
Nov 03 03:37:54.615282 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:54.615254    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e23c501-f20c-4d5b-8946-410c65498dfb-openshift-service-ca\") pod \"maas-api-5bfdcdb597-ngddf\" (UID: \"3e23c501-f20c-4d5b-8946-410c65498dfb\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-ngddf"
Nov 03 03:37:54.628516 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:54.628486    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfdcdb597-ngddf"]
Nov 03 03:37:54.716818 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:54.716782    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82psw\" (UniqueName: \"kubernetes.io/projected/3e23c501-f20c-4d5b-8946-410c65498dfb-kube-api-access-82psw\") pod \"maas-api-5bfdcdb597-ngddf\" (UID: \"3e23c501-f20c-4d5b-8946-410c65498dfb\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-ngddf"
Nov 03 03:37:54.717005 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:54.716830    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e23c501-f20c-4d5b-8946-410c65498dfb-openshift-service-ca\") pod \"maas-api-5bfdcdb597-ngddf\" (UID: \"3e23c501-f20c-4d5b-8946-410c65498dfb\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-ngddf"
Nov 03 03:37:54.717005 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:54.716856    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e23c501-f20c-4d5b-8946-410c65498dfb-metrics-certs\") pod \"maas-api-5bfdcdb597-ngddf\" (UID: \"3e23c501-f20c-4d5b-8946-410c65498dfb\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-ngddf"
Nov 03 03:37:54.717005 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:54.716923    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e23c501-f20c-4d5b-8946-410c65498dfb-maas-api-tls\") pod \"maas-api-5bfdcdb597-ngddf\" (UID: \"3e23c501-f20c-4d5b-8946-410c65498dfb\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-ngddf"
Nov 03 03:37:54.717522 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:54.717498    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e23c501-f20c-4d5b-8946-410c65498dfb-openshift-service-ca\") pod \"maas-api-5bfdcdb597-ngddf\" (UID: \"3e23c501-f20c-4d5b-8946-410c65498dfb\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-ngddf"
Nov 03 03:37:54.719576 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:54.719549    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e23c501-f20c-4d5b-8946-410c65498dfb-maas-api-tls\") pod \"maas-api-5bfdcdb597-ngddf\" (UID: \"3e23c501-f20c-4d5b-8946-410c65498dfb\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-ngddf"
Nov 03 03:37:54.719576 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:54.719570    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e23c501-f20c-4d5b-8946-410c65498dfb-metrics-certs\") pod \"maas-api-5bfdcdb597-ngddf\" (UID: \"3e23c501-f20c-4d5b-8946-410c65498dfb\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-ngddf"
Nov 03 03:37:54.770597 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:54.770563    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82psw\" (UniqueName: \"kubernetes.io/projected/3e23c501-f20c-4d5b-8946-410c65498dfb-kube-api-access-82psw\") pod \"maas-api-5bfdcdb597-ngddf\" (UID: \"3e23c501-f20c-4d5b-8946-410c65498dfb\") " pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-ngddf"
Nov 03 03:37:54.842011 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:54.841918    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-ngddf"
Nov 03 03:37:55.214983 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:55.214889    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfdcdb597-ngddf"]
Nov 03 03:37:55.221635 ip-10-0-139-26 kubenswrapper[5005]: W1103 03:37:55.221606    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e23c501_f20c_4d5b_8946_410c65498dfb.slice/crio-9e5533f18fd2a6f3f5e5bbed71d46ae3bd81ac0abf2e86aa2e761751e15e8a29 WatchSource:0}: Error finding container 9e5533f18fd2a6f3f5e5bbed71d46ae3bd81ac0abf2e86aa2e761751e15e8a29: Status 404 returned error can't find the container with id 9e5533f18fd2a6f3f5e5bbed71d46ae3bd81ac0abf2e86aa2e761751e15e8a29
Nov 03 03:37:55.223156 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:55.223121    5005 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 03:37:55.382141 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:55.382101    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-ngddf" event={"ID":"3e23c501-f20c-4d5b-8946-410c65498dfb","Type":"ContainerStarted","Data":"9e5533f18fd2a6f3f5e5bbed71d46ae3bd81ac0abf2e86aa2e761751e15e8a29"}
Nov 03 03:37:56.388657 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:56.388563    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-ngddf" event={"ID":"3e23c501-f20c-4d5b-8946-410c65498dfb","Type":"ContainerStarted","Data":"77c766941af69671ecdf4b7ee94afb07e165bb29def19a57f1101a22ce5408f6"}
Nov 03 03:37:56.388657 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:56.388629    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-ngddf"
Nov 03 03:37:56.497610 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:37:56.497554    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-ngddf" podStartSLOduration=1.64896163 podStartE2EDuration="2.497539629s" totalImagesPullingTime="848.577999ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:37:56.496043921 +0000 UTC m=+6403.510285291" watchObservedRunningTime="2026-11-03 03:37:56.497539629 +0000 UTC m=+6403.511780995"
Nov 03 03:38:00.773851 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:38:00.773810    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:38:02.400249 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:02.400215    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5bfdcdb597-ngddf"
Nov 03 03:38:02.558893 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:02.558863    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"]
Nov 03 03:38:02.559094 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:02.559071    5005 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f" podUID="87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce" containerName="maas-api" containerID="cri-o://b4b89f7c010ec494b3d6c7c9b9e3a43fc5abd3829a381d83d6f0a89e6b640776" gracePeriod=30
Nov 03 03:38:03.149345 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:03.149299    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f" podUID="87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.51:8443/health\": dial tcp 10.132.0.51:8443: connect: connection refused"
Nov 03 03:38:03.821454 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:03.821429    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"
Nov 03 03:38:03.921090 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:03.921004    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-kube-api-access-7jwrd\" (UniqueName: \"kubernetes.io/projected/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-kube-api-access-7jwrd\") pod \"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce\" (UID: \"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce\") "
Nov 03 03:38:03.921090 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:03.921042    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-openshift-service-ca\") pod \"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce\" (UID: \"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce\") "
Nov 03 03:38:03.921090 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:03.921081    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-metrics-certs\") pod \"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce\" (UID: \"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce\") "
Nov 03 03:38:03.921395 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:03.921258    5005 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-maas-api-tls\") pod \"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce\" (UID: \"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce\") "
Nov 03 03:38:03.921505 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:03.921468    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-openshift-service-ca" pod "87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce" (UID: "87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:38:03.921805 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:03.921785    5005 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-openshift-service-ca\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:38:03.923587 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:03.923555    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-maas-api-tls" pod "87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce" (UID: "87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:38:03.923842 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:03.923818    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-metrics-certs" pod "87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce" (UID: "87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:38:03.924172 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:03.924150    5005 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-kube-api-access-7jwrd" pod "87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce" (UID: "87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce"). InnerVolumeSpecName "kube-api-access-7jwrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:38:04.023176 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:04.023136    5005 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jwrd\" (UniqueName: \"kubernetes.io/projected/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-kube-api-access-7jwrd\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:38:04.023176 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:04.023172    5005 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-metrics-certs\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:38:04.023176 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:04.023182    5005 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce-maas-api-tls\") on node \"ip-10-0-139-26.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:38:04.482203 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:04.482164    5005 generic.go:356] "Generic (PLEG): container finished" podID="87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce" containerID="b4b89f7c010ec494b3d6c7c9b9e3a43fc5abd3829a381d83d6f0a89e6b640776" exitCode=0
Nov 03 03:38:04.482397 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:04.482226    5005 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"
Nov 03 03:38:04.482397 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:04.482249    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f" event={"ID":"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce","Type":"ContainerDied","Data":"b4b89f7c010ec494b3d6c7c9b9e3a43fc5abd3829a381d83d6f0a89e6b640776"}
Nov 03 03:38:04.482397 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:04.482304    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f" event={"ID":"87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce","Type":"ContainerDied","Data":"99c227d903131718eea5984b0fe655be80414cdf402d6b676ed83acf58982b42"}
Nov 03 03:38:04.482397 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:04.482326    5005 scope.go:122] "RemoveContainer" containerID="b4b89f7c010ec494b3d6c7c9b9e3a43fc5abd3829a381d83d6f0a89e6b640776"
Nov 03 03:38:04.492506 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:04.492485    5005 scope.go:122] "RemoveContainer" containerID="b4b89f7c010ec494b3d6c7c9b9e3a43fc5abd3829a381d83d6f0a89e6b640776"
Nov 03 03:38:04.492853 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:38:04.492835    5005 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4b89f7c010ec494b3d6c7c9b9e3a43fc5abd3829a381d83d6f0a89e6b640776\": container with ID starting with b4b89f7c010ec494b3d6c7c9b9e3a43fc5abd3829a381d83d6f0a89e6b640776 not found: ID does not exist" containerID="b4b89f7c010ec494b3d6c7c9b9e3a43fc5abd3829a381d83d6f0a89e6b640776"
Nov 03 03:38:04.492904 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:04.492863    5005 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b89f7c010ec494b3d6c7c9b9e3a43fc5abd3829a381d83d6f0a89e6b640776"} err="failed to get container status \"b4b89f7c010ec494b3d6c7c9b9e3a43fc5abd3829a381d83d6f0a89e6b640776\": rpc error: code = NotFound desc = could not find container \"b4b89f7c010ec494b3d6c7c9b9e3a43fc5abd3829a381d83d6f0a89e6b640776\": container with ID starting with b4b89f7c010ec494b3d6c7c9b9e3a43fc5abd3829a381d83d6f0a89e6b640776 not found: ID does not exist"
Nov 03 03:38:04.636290 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:04.636225    5005 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"]
Nov 03 03:38:04.682715 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:04.682667    5005 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-f9b4c8d5c-rz97f"]
Nov 03 03:38:05.111606 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:38:05.111545    5005 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce" path="/var/lib/kubelet/pods/87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce/volumes"
Nov 03 03:38:30.712822 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:38:30.712736    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:39:00.714093 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:39:00.714051    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:39:30.713032 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:39:30.712988    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:40:00.713810 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:40:00.713771    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:40:12.534855 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:40:12.534805    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-vpjhq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:40:12.535611 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:40:12.535575    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-vpjhq" podUID="239e8691-a4ce-4e72-ac2d-5dbc72b182d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:40:28.595888 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:40:28.595844    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-pd8pt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:40:28.596404 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:40:28.595909    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt" podUID="ee540784-ffc0-40e2-be04-d8c13a2521b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:40:30.713503 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:40:30.713461    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:41:00.772505 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:41:00.772427    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:41:14.056614 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:41:14.056482    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 03:41:14.086829 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:41:14.086805    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-rjmbd_a8186db5-7ed8-4fd8-95e8-ecd2fd7087fd/console-operator/2.log"
Nov 03 03:41:28.596992 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:41:28.596901    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-pd8pt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:41:28.596992 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:41:28.596971    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt" podUID="ee540784-ffc0-40e2-be04-d8c13a2521b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:41:30.714078 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:41:30.714032    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:42:00.713637 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:42:00.713594    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:42:30.717112 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:42:30.717075    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:43:00.714070 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:43:00.714029    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:43:28.595902 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:43:28.595856    5005 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-pd8pt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:43:28.596420 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:43:28.595938    5005 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-pd8pt" podUID="ee540784-ffc0-40e2-be04-d8c13a2521b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:43:30.714170 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:43:30.714127    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:44:00.713481 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:44:00.713436    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:44:16.883503 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:44:16.883468    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-n4klg_e9e336ab-f712-4f2c-b897-fc298e3bfb99/manager/0.log"
Nov 03 03:44:18.399192 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:44:18.399164    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-kzmvf_f5a529b5-98e0-433d-ad13-382685e55c5f/server/0.log"
Nov 03 03:44:18.614840 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:44:18.614811    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kzgz4_a82e336a-5128-4b44-add6-ea01d6179258/manager/0.log"
Nov 03 03:44:19.028860 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:44:19.028828    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kflsp_e3a46aa4-c8f9-4fdd-a80a-19f100584149/proxy/0.log"
Nov 03 03:44:20.419521 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:44:20.419489    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5bfdcdb597-ngddf_3e23c501-f20c-4d5b-8946-410c65498dfb/maas-api/0.log"
Nov 03 03:44:24.297439 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:44:24.297351    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-nv5dz_4fabfb92-7556-4a30-a90f-8419b2b2ad95/postgres/0.log"
Nov 03 03:44:27.107984 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:44:27.107957    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-sxxkw_75f9fa81-69b8-4cfe-acdb-8e68d6e93d02/kuadrant-console-plugin/0.log"
Nov 03 03:44:29.339605 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:44:29.339575    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-4rgnn_c36dba16-afb6-4239-aef1-4c3dfa4fa43d/discovery/0.log"
Nov 03 03:44:29.689619 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:44:29.689543    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dc8f585b4-g6dbz_d3b1e1c0-194f-47a1-8a7b-7ab2ff3dbf2d/kube-auth-proxy/0.log"
Nov 03 03:44:30.714314 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:44:30.714274    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:44:31.276340 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:44:31.276310    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-f72x6_32593158-9c57-4ecc-8a47-a7ca894ed169/istio-proxy/0.log"
Nov 03 03:44:32.179409 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:44:32.179375    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-pd8pt_ee540784-ffc0-40e2-be04-d8c13a2521b1/istio-proxy/0.log"
Nov 03 03:44:32.852783 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:44:32.852747    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-vpjhq_239e8691-a4ce-4e72-ac2d-5dbc72b182d7/istio-proxy/0.log"
Nov 03 03:44:33.121402 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:44:33.121311    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4_c7ef5e81-eb41-4e31-a4d3-437e40ea8778/istio-proxy/0.log"
Nov 03 03:44:33.798320 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:44:33.798287    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7875dd7849-ptjkr_7a9e57fd-a6e6-4ddb-aa08-ac1390403544/router/0.log"
Nov 03 03:44:34.943655 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:44:34.943623    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-sc5zm_ffefebf4-2c90-428e-a018-543169ba806c/main/0.log"
Nov 03 03:44:35.471704 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:44:35.471665    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-2dzf7_91c8d98e-c78f-4749-8a24-86d28672ddbb/main/0.log"
Nov 03 03:44:36.186708 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:44:36.186672    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-44tnl_cff8b24c-e3ef-4500-859c-528ffdf39d1b/main/0.log"
Nov 03 03:44:36.686079 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:44:36.686047    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-hf5cr_96c17dd0-a778-4d4d-851f-551c152453ef/main/0.log"
Nov 03 03:44:57.223805 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:44:57.223768    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mxg52_79fe02fa-99ce-4222-b569-6b323df47ab4/global-pull-secret-syncer/0.log"
Nov 03 03:44:58.374559 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:44:58.374529    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-jqkfr_e11ea02d-ac52-4318-9fdd-95c0bb171aed/connection-checker/0.log"
Nov 03 03:44:58.766797 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:44:58.766771    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-l87nq_38de413a-873a-4288-85f5-c453581a53d6/connection-checker/0.log"
Nov 03 03:44:59.158718 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:44:59.158624    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-rcv8q_3a6655a6-9079-4927-8e6c-09d075de438c/connection-checker/0.log"
Nov 03 03:45:00.602431 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:00.602397    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vvzws_1f0923bb-db25-4266-ad8d-2e1cfb7f602b/konnectivity-agent/0.log"
Nov 03 03:45:00.714486 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:45:00.714449    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"
Nov 03 03:45:01.526781 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:01.526747    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-26.us-west-2.compute.internal_37711de5beea2605bd7d0ac9a465c416/haproxy/0.log"
Nov 03 03:45:04.134031 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:04.134002    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-n4klg_e9e336ab-f712-4f2c-b897-fc298e3bfb99/manager/0.log"
Nov 03 03:45:05.911023 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:05.910991    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-kzmvf_f5a529b5-98e0-433d-ad13-382685e55c5f/server/0.log"
Nov 03 03:45:07.260726 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.260676    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kzgz4_a82e336a-5128-4b44-add6-ea01d6179258/manager/0.log"
Nov 03 03:45:07.376419 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.376388    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kflsp_e3a46aa4-c8f9-4fdd-a80a-19f100584149/proxy/0.log"
Nov 03 03:45:07.732141 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.732053    5005 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h"]
Nov 03 03:45:07.732729 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.732711    5005 cpu_manager.go:414] "containerMap: removing container" podUID="87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce" containerName="maas-api"
Nov 03 03:45:07.732790 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.732732    5005 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce" containerName="maas-api"
Nov 03 03:45:07.732916 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.732904    5005 memory_manager.go:365] "RemoveStaleState removing state" podUID="87dd2b4f-63ef-40be-bf4d-3d8e1f71d0ce" containerName="maas-api"
Nov 03 03:45:07.736550 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.736533    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h"
Nov 03 03:45:07.775515 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.775477    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/577c6e1d-8954-4a7b-bc6c-057fe72c56f8-sys\") pod \"perf-node-gather-daemonset-fh96h\" (UID: \"577c6e1d-8954-4a7b-bc6c-057fe72c56f8\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h"
Nov 03 03:45:07.775515 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.775515    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/577c6e1d-8954-4a7b-bc6c-057fe72c56f8-proc\") pod \"perf-node-gather-daemonset-fh96h\" (UID: \"577c6e1d-8954-4a7b-bc6c-057fe72c56f8\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h"
Nov 03 03:45:07.775759 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.775534    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/577c6e1d-8954-4a7b-bc6c-057fe72c56f8-lib-modules\") pod \"perf-node-gather-daemonset-fh96h\" (UID: \"577c6e1d-8954-4a7b-bc6c-057fe72c56f8\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h"
Nov 03 03:45:07.775759 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.775594    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlf5m\" (UniqueName: \"kubernetes.io/projected/577c6e1d-8954-4a7b-bc6c-057fe72c56f8-kube-api-access-xlf5m\") pod \"perf-node-gather-daemonset-fh96h\" (UID: \"577c6e1d-8954-4a7b-bc6c-057fe72c56f8\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h"
Nov 03 03:45:07.775759 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.775728    5005 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/577c6e1d-8954-4a7b-bc6c-057fe72c56f8-podres\") pod \"perf-node-gather-daemonset-fh96h\" (UID: \"577c6e1d-8954-4a7b-bc6c-057fe72c56f8\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h"
Nov 03 03:45:07.785198 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.785141    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l75xk\"/\"openshift-service-ca.crt\""
Nov 03 03:45:07.785391 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.785276    5005 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l75xk\"/\"kube-root-ca.crt\""
Nov 03 03:45:07.831996 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.831964    5005 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-l75xk\"/\"default-dockercfg-hq8x6\""
Nov 03 03:45:07.835201 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.835175    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h"]
Nov 03 03:45:07.876387 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.876348    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/577c6e1d-8954-4a7b-bc6c-057fe72c56f8-sys\") pod \"perf-node-gather-daemonset-fh96h\" (UID: \"577c6e1d-8954-4a7b-bc6c-057fe72c56f8\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h"
Nov 03 03:45:07.876560 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.876395    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/577c6e1d-8954-4a7b-bc6c-057fe72c56f8-proc\") pod \"perf-node-gather-daemonset-fh96h\" (UID: \"577c6e1d-8954-4a7b-bc6c-057fe72c56f8\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h"
Nov 03 03:45:07.876560 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.876482    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/577c6e1d-8954-4a7b-bc6c-057fe72c56f8-sys\") pod \"perf-node-gather-daemonset-fh96h\" (UID: \"577c6e1d-8954-4a7b-bc6c-057fe72c56f8\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h"
Nov 03 03:45:07.876560 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.876501    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/577c6e1d-8954-4a7b-bc6c-057fe72c56f8-lib-modules\") pod \"perf-node-gather-daemonset-fh96h\" (UID: \"577c6e1d-8954-4a7b-bc6c-057fe72c56f8\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h"
Nov 03 03:45:07.876670 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.876545    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/577c6e1d-8954-4a7b-bc6c-057fe72c56f8-proc\") pod \"perf-node-gather-daemonset-fh96h\" (UID: \"577c6e1d-8954-4a7b-bc6c-057fe72c56f8\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h"
Nov 03 03:45:07.876670 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.876581    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/577c6e1d-8954-4a7b-bc6c-057fe72c56f8-lib-modules\") pod \"perf-node-gather-daemonset-fh96h\" (UID: \"577c6e1d-8954-4a7b-bc6c-057fe72c56f8\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h"
Nov 03 03:45:07.876670 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.876615    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlf5m\" (UniqueName: \"kubernetes.io/projected/577c6e1d-8954-4a7b-bc6c-057fe72c56f8-kube-api-access-xlf5m\") pod \"perf-node-gather-daemonset-fh96h\" (UID: \"577c6e1d-8954-4a7b-bc6c-057fe72c56f8\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h"
Nov 03 03:45:07.876670 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.876662    5005 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/577c6e1d-8954-4a7b-bc6c-057fe72c56f8-podres\") pod \"perf-node-gather-daemonset-fh96h\" (UID: \"577c6e1d-8954-4a7b-bc6c-057fe72c56f8\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h"
Nov 03 03:45:07.876853 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.876833    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/577c6e1d-8954-4a7b-bc6c-057fe72c56f8-podres\") pod \"perf-node-gather-daemonset-fh96h\" (UID: \"577c6e1d-8954-4a7b-bc6c-057fe72c56f8\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h"
Nov 03 03:45:07.930941 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:07.930908    5005 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlf5m\" (UniqueName: \"kubernetes.io/projected/577c6e1d-8954-4a7b-bc6c-057fe72c56f8-kube-api-access-xlf5m\") pod \"perf-node-gather-daemonset-fh96h\" (UID: \"577c6e1d-8954-4a7b-bc6c-057fe72c56f8\") " pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h"
Nov 03 03:45:08.049907 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:08.049871    5005 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h"
Nov 03 03:45:08.422261 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:08.422184    5005 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h"]
Nov 03 03:45:08.429298 ip-10-0-139-26 kubenswrapper[5005]: W1103 03:45:08.429262    5005 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod577c6e1d_8954_4a7b_bc6c_057fe72c56f8.slice/crio-a7e4cea68f0183e2ff59942183599f4d9162453ba735642d3203745253f6edc0 WatchSource:0}: Error finding container a7e4cea68f0183e2ff59942183599f4d9162453ba735642d3203745253f6edc0: Status 404 returned error can't find the container with id a7e4cea68f0183e2ff59942183599f4d9162453ba735642d3203745253f6edc0
Nov 03 03:45:08.765920 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:08.765893    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-sxxkw_75f9fa81-69b8-4cfe-acdb-8e68d6e93d02/kuadrant-console-plugin/0.log"
Nov 03 03:45:09.426552 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:09.426516    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h" event={"ID":"577c6e1d-8954-4a7b-bc6c-057fe72c56f8","Type":"ContainerStarted","Data":"b1714cc7a77ba239528fd0adfd7ef128a06f086e85f6987afc3881f97544abe7"}
Nov 03 03:45:09.427040 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:09.426571    5005 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h" event={"ID":"577c6e1d-8954-4a7b-bc6c-057fe72c56f8","Type":"ContainerStarted","Data":"a7e4cea68f0183e2ff59942183599f4d9162453ba735642d3203745253f6edc0"}
Nov 03 03:45:09.427040 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:09.426642    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h"
Nov 03 03:45:09.534123 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:09.534072    5005 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h" podStartSLOduration=2.5340599729999997 podStartE2EDuration="2.534059973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:45:09.533624162 +0000 UTC m=+6836.547865528" watchObservedRunningTime="2026-11-03 03:45:09.534059973 +0000 UTC m=+6836.548301338"
Nov 03 03:45:10.394333 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:10.394300    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-n4klg_e9e336ab-f712-4f2c-b897-fc298e3bfb99/manager/0.log"
Nov 03 03:45:11.992081 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:11.992051    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-kzmvf_f5a529b5-98e0-433d-ad13-382685e55c5f/server/0.log"
Nov 03 03:45:13.189814 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:13.189781    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kzgz4_a82e336a-5128-4b44-add6-ea01d6179258/manager/0.log"
Nov 03 03:45:13.361176 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:13.361143    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kflsp_e3a46aa4-c8f9-4fdd-a80a-19f100584149/proxy/0.log"
Nov 03 03:45:15.443993 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:15.443957    5005 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-l75xk/perf-node-gather-daemonset-fh96h"
Nov 03 03:45:15.639383 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:15.639348    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-g8wz8_f4425c17-5cc6-4afc-922c-d74694805196/cluster-monitoring-operator/0.log"
Nov 03 03:45:16.460538 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:16.460499    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-n4klg_e9e336ab-f712-4f2c-b897-fc298e3bfb99/manager/0.log"
Nov 03 03:45:18.191076 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:18.191047    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-kzmvf_f5a529b5-98e0-433d-ad13-382685e55c5f/server/0.log"
Nov 03 03:45:19.311435 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:19.311398    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kzgz4_a82e336a-5128-4b44-add6-ea01d6179258/manager/0.log"
Nov 03 03:45:19.475396 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:19.475362    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kflsp_e3a46aa4-c8f9-4fdd-a80a-19f100584149/proxy/0.log"
Nov 03 03:45:20.661674 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:20.661636    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xl2vn_069f1d59-3556-40b1-9769-f137ef715740/node-exporter/0.log"
Nov 03 03:45:21.067550 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:21.067523    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xl2vn_069f1d59-3556-40b1-9769-f137ef715740/kube-rbac-proxy/0.log"
Nov 03 03:45:21.473197 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:21.473123    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xl2vn_069f1d59-3556-40b1-9769-f137ef715740/init-textfile/0.log"
Nov 03 03:45:25.883945 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:25.883913    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-4rgnn_c36dba16-afb6-4239-aef1-4c3dfa4fa43d/discovery/0.log"
Nov 03 03:45:25.991253 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:25.991227    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dc8f585b4-g6dbz_d3b1e1c0-194f-47a1-8a7b-7ab2ff3dbf2d/kube-auth-proxy/0.log"
Nov 03 03:45:26.810363 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:26.810331    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-2jlbf_5e41d0c6-c6ea-4033-a362-98fca79bdd39/prometheus-operator-admission-webhook/0.log"
Nov 03 03:45:27.245096 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:27.245018    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-f72x6_32593158-9c57-4ecc-8a47-a7ca894ed169/istio-proxy/0.log"
Nov 03 03:45:27.873735 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:27.873686    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-pd8pt_ee540784-ffc0-40e2-be04-d8c13a2521b1/istio-proxy/0.log"
Nov 03 03:45:28.289794 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:28.289763    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-vpjhq_239e8691-a4ce-4e72-ac2d-5dbc72b182d7/istio-proxy/0.log"
Nov 03 03:45:28.496584 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:28.496555    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64695b4bc9-llpk4_c7ef5e81-eb41-4e31-a4d3-437e40ea8778/istio-proxy/0.log"
Nov 03 03:45:29.125902 ip-10-0-139-26 kubenswrapper[5005]: I1103 03:45:29.125873    5005 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7875dd7849-ptjkr_7a9e57fd-a6e6-4ddb-aa08-ac1390403544/router/0.log"
Nov 03 03:45:30.714882 ip-10-0-139-26 kubenswrapper[5005]: E1103 03:45:30.714843    5005 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9twrb" containerName="extractor"