Oct 30 20:18:59.894663 ip-10-0-138-34 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 20:19:00.267721 ip-10-0-138-34 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:19:00.267721 ip-10-0-138-34 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 20:19:00.267721 ip-10-0-138-34 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:19:00.267721 ip-10-0-138-34 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:19:00.272512 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272418    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:00.272512 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272508    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:00.272512 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272513    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:00.272512 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272515    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:00.272512 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272518    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:00.272512 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272521    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:00.272742 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272524    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:00.272742 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272527    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:00.272742 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272529    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:00.272742 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272532    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:00.272742 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272534    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:00.272742 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272537    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:00.272742 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272539    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:00.272742 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272542    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:00.272742 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272545    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:00.272742 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272548    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:00.272742 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272552    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:00.272742 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272557    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:00.272742 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272560    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:00.272742 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272563    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:00.272742 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272565    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:00.272742 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272568    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:00.272742 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272571    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:00.272742 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272574    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:00.272742 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272576    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:00.273161 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272579    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:00.273161 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272581    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:00.273161 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272584    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:00.273161 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272594    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:00.273161 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272597    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:00.273161 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272599    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:00.273161 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272602    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:00.273161 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272604    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:00.273161 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272607    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:00.273161 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272609    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:00.273161 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272611    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:00.273161 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272614    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:00.273161 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272616    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:00.273161 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272619    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:00.273161 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272622    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:00.273161 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272625    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:00.273161 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272628    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:00.273161 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272630    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:00.273161 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272633    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:00.273161 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272636    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:00.273641 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272638    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:00.273641 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272640    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:00.273641 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272643    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:00.273641 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272645    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:00.273641 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272648    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:00.273641 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272650    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:00.273641 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272653    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:00.273641 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272655    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:00.273641 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272657    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:00.273641 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272661    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:00.273641 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272663    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:00.273641 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272666    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:00.273641 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272668    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:00.273641 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272671    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:00.273641 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272673    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:00.273641 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272676    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:00.273641 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272678    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:00.273641 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272681    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:00.273641 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272683    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:00.274082 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272685    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:00.274082 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272688    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:00.274082 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272690    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:00.274082 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272693    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:00.274082 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272697    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:00.274082 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272700    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:00.274082 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272705    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:00.274082 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272709    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:00.274082 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272712    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:00.274082 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272715    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:00.274082 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272718    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:00.274082 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272721    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:00.274082 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272723    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:00.274082 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272726    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:00.274082 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272728    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:00.274082 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272730    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:00.274082 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272733    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:00.274082 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272735    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:00.274082 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272737    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:00.274082 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272741    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:00.274549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272744    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:00.274549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272747    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:00.274549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272749    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:00.274549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272752    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:00.274549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272755    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:00.274549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272757    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:00.274549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272760    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:00.274549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272762    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:00.274549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272765    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:00.274549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272767    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:00.274549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272769    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:00.274549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272772    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:00.274549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272775    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:00.274549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272777    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:00.274549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272779    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:00.274549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272782    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:00.274549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272785    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:00.274549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272787    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:00.274549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272790    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:00.274549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272792    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:00.275009 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272795    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:00.275009 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.272797    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:00.276195 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276181    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:00.276195 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276196    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:00.276252 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276200    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:00.276252 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276204    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:00.276252 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276207    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:00.276252 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276209    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:00.276252 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276212    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:00.276252 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276215    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:00.276252 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276217    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:00.276252 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276220    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:00.276252 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276223    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:00.276252 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276226    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:00.276252 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276228    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:00.276252 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276231    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:00.276252 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276233    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:00.276252 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276236    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:00.276252 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276238    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:00.276252 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276240    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:00.276252 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276243    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:00.276252 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276245    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:00.276252 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276247    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:00.276252 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276250    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:00.276737 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276254    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:00.276737 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276258    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:00.276737 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276262    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:00.276737 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276265    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:00.276737 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276267    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:00.276737 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276270    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:00.276737 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276272    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:00.276737 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276275    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:00.276737 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276277    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:00.276737 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276280    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:00.276737 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276283    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:00.276737 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276285    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:00.276737 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276287    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:00.276737 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276290    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:00.276737 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276292    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:00.276737 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276294    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:00.276737 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276297    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:00.276737 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276299    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:00.276737 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276301    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:00.276737 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276304    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:00.277200 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276306    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:00.277200 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276309    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:00.277200 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276311    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:00.277200 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276314    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:00.277200 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276316    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:00.277200 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276332    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:00.277200 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276335    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:00.277200 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276338    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:00.277200 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276341    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:00.277200 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276344    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:00.277200 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276347    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:00.277200 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276349    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:00.277200 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276352    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:00.277200 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276355    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:00.277200 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276357    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:00.277200 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276360    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:00.277200 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276362    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:00.277200 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276365    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:00.277200 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276367    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:00.277200 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276370    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:00.277703 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276373    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:00.277703 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276375    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:00.277703 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276377    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:00.277703 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276380    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:00.277703 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276382    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:00.277703 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276384    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:00.277703 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276387    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:00.277703 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276389    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:00.277703 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276392    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:00.277703 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276394    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:00.277703 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276396    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:00.277703 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276399    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:00.277703 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276401    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:00.277703 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276403    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:00.277703 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276407    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:00.277703 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276409    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:00.277703 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276412    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:00.277703 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276414    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:00.277703 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276417    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:00.278131 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276419    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:00.278131 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276422    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:00.278131 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276425    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:00.278131 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276427    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:00.278131 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276429    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:00.278131 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276432    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:00.278131 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276435    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:00.278131 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276438    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:00.278131 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276440    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:00.278131 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276443    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:00.278131 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276445    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:00.278131 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276447    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:00.278131 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276450    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:00.278131 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276452    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:00.278131 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276455    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:00.278131 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276457    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:00.278131 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276461    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:00.278131 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276465    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:00.278131 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276467    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:00.278607 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276470    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:00.278607 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276473    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:00.278607 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276476    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:00.278607 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276478    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:00.278607 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276481    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:00.278607 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276483    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:00.278607 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276991    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:00.278607 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276995    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:00.278607 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.276999    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:00.278607 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277002    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:00.278607 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277005    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:00.278607 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277009    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:00.278607 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277011    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:00.278607 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277013    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:00.278607 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277016    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:00.278607 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277019    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:00.278607 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277021    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:00.278607 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277024    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:00.278607 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277026    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:00.278607 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277029    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:00.279075 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277031    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:00.279075 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277033    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:00.279075 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277036    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:00.279075 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277038    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:00.279075 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277041    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:00.279075 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277043    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:00.279075 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277046    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:00.279075 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277048    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:00.279075 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277050    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:00.279075 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277053    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:00.279075 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277055    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:00.279075 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277058    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:00.279075 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277062    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:00.279075 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277065    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:00.279075 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277067    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:00.279075 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277070    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:00.279075 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277072    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:00.279075 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277075    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:00.279075 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277077    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:00.279075 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277080    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:00.279543 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277082    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:00.279543 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277085    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:00.279543 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277088    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:00.279543 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277091    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:00.279543 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277093    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:00.279543 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277096    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:00.279543 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277100    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:00.279543 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277104    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:00.279543 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277107    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:00.279543 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277109    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:00.279543 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277111    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:00.279543 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277114    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:00.279543 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277116    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:00.279543 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277119    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:00.279543 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277121    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:00.279543 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277124    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:00.279543 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277126    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:00.279543 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277129    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:00.279543 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277131    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:00.280009 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277133    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:00.280009 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277136    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:00.280009 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277138    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:00.280009 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277140    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:00.280009 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277143    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:00.280009 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277147    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:00.280009 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277149    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:00.280009 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277152    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:00.280009 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277154    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:00.280009 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277157    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:00.280009 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277159    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:00.280009 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277162    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:00.280009 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277164    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:00.280009 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277167    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:00.280009 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277169    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:00.280009 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277172    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:00.280009 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277175    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:00.280009 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277177    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:00.280009 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277180    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:00.280458 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277184    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:00.280458 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277187    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:00.280458 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277191    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:00.280458 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277193    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:00.280458 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277196    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:00.280458 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277199    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:00.280458 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277201    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:00.280458 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277204    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:00.280458 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277206    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:00.280458 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277209    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:00.280458 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277212    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:00.280458 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277214    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:00.280458 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277217    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:00.280458 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277219    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:00.280458 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277221    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:00.280458 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277224    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:00.280458 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277226    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:00.280458 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277228    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:00.280458 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277231    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:00.280883 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277233    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:00.280883 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277236    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:00.280883 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277239    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:00.280883 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277241    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:00.280883 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277243    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:00.280883 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277246    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:00.280883 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277248    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:00.280883 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277250    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:00.280883 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277253    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:00.280883 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277255    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:00.280883 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277258    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:00.280883 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277260    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:00.280883 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277263    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:00.280883 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277265    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:00.280883 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.277267    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:00.280883 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277379    2577 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 20:19:00.280883 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277389    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 20:19:00.280883 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277400    2577 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 20:19:00.280883 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277404    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 20:19:00.280883 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277408    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 20:19:00.280883 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277412    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277420    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277425    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277428    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277431    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277434    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277437    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277441    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277444    2577 flags.go:64] FLAG: --cgroup-root=""
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277446    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277449    2577 flags.go:64] FLAG: --client-ca-file=""
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277452    2577 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277454    2577 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277459    2577 flags.go:64] FLAG: --cluster-domain=""
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277462    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277465    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277467    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277471    2577 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277475    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277478    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277481    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277484    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277489    2577 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277493    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 20:19:00.281428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277496    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277499    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277501    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277506    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277509    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277512    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277514    2577 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277517    2577 flags.go:64] FLAG: --enable-server="true"
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277520    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277524    2577 flags.go:64] FLAG: --event-burst="100"
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277527    2577 flags.go:64] FLAG: --event-qps="50"
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277530    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277533    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277536    2577 flags.go:64] FLAG: --eviction-hard=""
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277540    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277543    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277546    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277548    2577 flags.go:64] FLAG: --eviction-soft=""
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277551    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277554    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277557    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277560    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277563    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277566    2577 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277568    2577 flags.go:64] FLAG: --feature-gates=""
Oct 30 20:19:00.281986 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277572    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277575    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277578    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277581    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277584    2577 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277588    2577 flags.go:64] FLAG: --help="false"
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277591    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277594    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277597    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277600    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277603    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277607    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277609    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277612    2577 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277615    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277618    2577 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277620    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277624    2577 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277626    2577 flags.go:64] FLAG: --kube-reserved=""
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277629    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277632    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277635    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277637    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 20:19:00.282557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277640    2577 flags.go:64] FLAG: --lock-file=""
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277643    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277646    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277649    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277654    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277657    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277659    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277662    2577 flags.go:64] FLAG: --logging-format="text"
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277665    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277668    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277671    2577 flags.go:64] FLAG: --manifest-url=""
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277673    2577 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277678    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277681    2577 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277686    2577 flags.go:64] FLAG: --max-pods="110"
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277689    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277691    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277695    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277698    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277701    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277704    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277706    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277714    2577 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277717    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277720    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 20:19:00.283087 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277723    2577 flags.go:64] FLAG: --pod-cidr=""
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277726    2577 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277730    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277733    2577 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277736    2577 flags.go:64] FLAG: --port="10250"
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277739    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277741    2577 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-00b0b70f5f93b92b2"
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277745    2577 flags.go:64] FLAG: --qos-reserved=""
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277747    2577 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277750    2577 flags.go:64] FLAG: --register-node="true"
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277753    2577 flags.go:64] FLAG: --register-with-taints=""
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277761    2577 flags.go:64] FLAG: --registry-burst="10"
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277764    2577 flags.go:64] FLAG: --registry-qps="5"
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277766    2577 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277769    2577 flags.go:64] FLAG: --reserved-memory=""
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277773    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277776    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277779    2577 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277782    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277785    2577 flags.go:64] FLAG: --runonce="false"
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277788    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277792    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277795    2577 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277797    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277800    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277804    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 20:19:00.283650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277807    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277810    2577 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277813    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277816    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277818    2577 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277821    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277824    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277826    2577 flags.go:64] FLAG: --system-cgroups=""
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277830    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277836    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277839    2577 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277841    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277845    2577 flags.go:64] FLAG: --tls-min-version=""
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277848    2577 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277851    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277853    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277856    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277859    2577 flags.go:64] FLAG: --v="2"
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277863    2577 flags.go:64] FLAG: --version="false"
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277867    2577 flags.go:64] FLAG: --vmodule=""
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277871    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.277874    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278011    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278014    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:00.284278 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278017    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:00.284825 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278019    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:00.284825 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278022    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:00.284825 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278026    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:00.284825 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278029    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:00.284825 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278031    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:00.284825 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278034    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:00.284825 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278037    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:00.284825 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278040    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:00.284825 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278043    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:00.284825 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278045    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:00.284825 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278048    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:00.284825 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278050    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:00.284825 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278052    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:00.284825 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278055    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:00.284825 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278057    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:00.284825 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278062    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:00.284825 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278064    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:00.284825 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278067    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:00.284825 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278070    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:00.284825 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278072    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:00.285275 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278074    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:00.285275 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278077    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:00.285275 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278079    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:00.285275 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278082    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:00.285275 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278084    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:00.285275 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278086    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:00.285275 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278089    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:00.285275 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278091    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:00.285275 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278094    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:00.285275 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278096    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:00.285275 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278099    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:00.285275 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278101    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:00.285275 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278104    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:00.285275 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278106    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:00.285275 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278110    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:00.285275 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278112    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:00.285275 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278114    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:00.285275 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278117    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:00.285275 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278119    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:00.285275 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278122    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:00.285755 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278125    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:00.285755 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278127    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:00.285755 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278130    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:00.285755 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278132    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:00.285755 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278135    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:00.285755 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278137    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:00.285755 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278140    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:00.285755 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278144    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:00.285755 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278146    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:00.285755 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278149    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:00.285755 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278151    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:00.285755 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278154    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:00.285755 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278156    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:00.285755 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278159    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:00.285755 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278161    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:00.285755 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278163    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:00.285755 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278166    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:00.285755 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278168    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:00.285755 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278171    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:00.285755 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278173    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:00.286211 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278175    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:00.286211 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278178    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:00.286211 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278180    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:00.286211 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278183    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:00.286211 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278185    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:00.286211 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278188    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:00.286211 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278191    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:00.286211 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278194    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:00.286211 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278196    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:00.286211 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278199    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:00.286211 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278201    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:00.286211 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278203    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:00.286211 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278206    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:00.286211 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278209    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:00.286211 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278211    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:00.286211 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278214    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:00.286211 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278216    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:00.286211 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278219    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:00.286211 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278221    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:00.286211 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278224    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:00.286729 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278227    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:00.286729 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278229    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:00.286729 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278232    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:00.286729 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278236    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:00.286729 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278240    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:00.286729 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278243    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:00.286729 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278245    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:00.286729 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278248    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:00.286729 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278251    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:00.286729 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278254    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:00.286729 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278258    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:00.286729 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278260    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:00.286729 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278264    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:00.286729 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278266    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:00.286729 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278269    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:00.286729 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278271    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:00.286729 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278273    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:00.286729 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278276    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:00.286729 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278279    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:00.287174 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278281    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:00.287174 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278284    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:00.287174 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278287    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:00.287174 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278289    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:00.287174 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.278315    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:19:00.287174 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278367    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:00.287174 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278372    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:00.287174 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278376    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:00.287174 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278379    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:00.287174 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278381    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:00.287174 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278384    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:00.287174 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278387    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:00.287174 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278389    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:00.287174 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278393    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:00.287174 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278395    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:00.287174 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278398    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:00.287601 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278400    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:00.287601 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278403    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:00.287601 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278405    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:00.287601 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278408    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:00.287601 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278410    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:00.287601 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278413    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:00.287601 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278415    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:00.287601 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278417    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:00.287601 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278420    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:00.287601 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278422    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:00.287601 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278424    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:00.287601 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278427    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:00.287601 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278429    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:00.287601 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278432    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:00.287601 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278434    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:00.287601 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278437    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:00.287601 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278440    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:00.287601 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278442    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:00.287601 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278445    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:00.287601 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278447    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:00.288059 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278449    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:00.288059 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278452    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:00.288059 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278455    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:00.288059 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278457    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:00.288059 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278460    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:00.288059 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278462    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:00.288059 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278465    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:00.288059 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278467    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:00.288059 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278470    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:00.288059 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278473    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:00.288059 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278475    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:00.288059 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278478    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:00.288059 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278480    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:00.288059 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278482    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:00.288059 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278485    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:00.288059 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278487    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:00.288059 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278489    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:00.288059 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278492    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:00.288059 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278494    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:00.288059 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278496    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:00.288530 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278499    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:00.288530 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278501    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:00.288530 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278503    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:00.288530 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278506    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:00.288530 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278508    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:00.288530 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278510    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:00.288530 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278512    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:00.288530 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278516    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:00.288530 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278519    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:00.288530 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278521    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:00.288530 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278523    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:00.288530 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278526    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:00.288530 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278528    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:00.288530 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278530    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:00.288530 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278533    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:00.288530 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278535    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:00.288530 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278538    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:00.288530 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278540    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:00.288530 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278543    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:00.288530 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278545    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:00.288984 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278548    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:00.288984 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278551    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:00.288984 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278554    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:00.288984 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278556    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:00.288984 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278559    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:00.288984 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278561    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:00.288984 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278563    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:00.288984 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278566    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:00.288984 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278568    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:00.288984 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278571    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:00.288984 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278573    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:00.288984 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278575    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:00.288984 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278579    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:00.288984 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278583    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:00.288984 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278587    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:00.288984 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278589    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:00.288984 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278592    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:00.288984 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278594    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:00.288984 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278598    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:00.289432 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278602    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:00.289432 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278605    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:00.289432 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278607    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:00.289432 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278610    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:00.289432 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278612    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:00.289432 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278615    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:00.289432 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278617    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:00.289432 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278620    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:00.289432 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278622    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:00.289432 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278625    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:00.289432 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278627    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:00.289432 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278630    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:00.289432 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278633    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:00.289432 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278636    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:00.289432 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278640    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:00.289432 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.278643    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:00.289432 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.286228    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 20:19:00.289432 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.286244    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 20:19:00.289432 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.286259    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 20:19:00.289890 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.286264    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 20:19:00.289890 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286347    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:00.289890 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286352    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:00.289890 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286356    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:00.289890 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286359    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:00.289890 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286361    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:00.289890 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286364    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:00.289890 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286366    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:00.289890 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286369    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:00.289890 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286372    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:00.289890 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286374    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:00.289890 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286376    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:00.289890 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286379    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:00.289890 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286382    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:00.289890 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286385    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:00.289890 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286387    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:00.289890 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286390    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:00.289890 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286393    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:00.289890 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286395    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:00.290340 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286397    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:00.290340 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286400    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:00.290340 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286403    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:00.290340 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286405    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:00.290340 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286408    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:00.290340 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286410    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:00.290340 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286412    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:00.290340 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286416    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:00.290340 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286420    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:00.290340 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286423    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:00.290340 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286427    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:00.290340 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286437    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:00.290340 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286440    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:00.290340 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286443    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:00.290340 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286445    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:00.290340 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286448    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:00.290340 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286450    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:00.290340 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286452    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:00.290340 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286455    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:00.290340 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286457    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:00.290790 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286460    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:00.290790 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286462    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:00.290790 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286464    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:00.290790 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286467    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:00.290790 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286469    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:00.290790 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286472    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:00.290790 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286474    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:00.290790 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286477    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:00.290790 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286480    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:00.290790 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286482    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:00.290790 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286484    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:00.290790 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286487    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:00.290790 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286489    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:00.290790 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286492    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:00.290790 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286494    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:00.290790 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286496    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:00.290790 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286499    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:00.290790 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286501    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:00.290790 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286504    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:00.291228 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286506    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:00.291228 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286508    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:00.291228 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286511    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:00.291228 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286514    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:00.291228 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286516    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:00.291228 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286524    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:00.291228 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286527    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:00.291228 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286529    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:00.291228 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286532    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:00.291228 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286534    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:00.291228 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286537    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:00.291228 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286539    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:00.291228 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286542    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:00.291228 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286544    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:00.291228 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286546    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:00.291228 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286549    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:00.291228 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286551    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:00.291228 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286553    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:00.291228 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286556    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:00.291228 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286558    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:00.291699 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286561    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:00.291699 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286564    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:00.291699 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286566    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:00.291699 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286569    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:00.291699 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286571    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:00.291699 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286573    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:00.291699 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286576    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:00.291699 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286578    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:00.291699 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286580    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:00.291699 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286583    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:00.291699 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286585    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:00.291699 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286587    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:00.291699 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286590    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:00.291699 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286592    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:00.291699 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286594    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:00.291699 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286597    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:00.291699 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286600    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:00.291699 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286602    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:00.291699 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286610    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:00.291699 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286613    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:00.292153 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286615    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:00.292153 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286617    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:00.292153 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286620    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:00.292153 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286622    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:00.292153 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286625    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:00.292153 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286627    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:00.292153 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286630    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:00.292153 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286632    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:00.292153 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286636    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:00.292153 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.286664    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:19:00.292153 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286842    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:00.292153 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286846    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:00.292153 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286849    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:00.292153 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286852    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:00.292153 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286855    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:00.292153 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286857    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:00.292541 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286860    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:00.292541 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286863    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:00.292541 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286865    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:00.292541 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286868    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:00.292541 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286870    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:00.292541 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286873    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:00.292541 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286875    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:00.292541 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286877    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:00.292541 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286879    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:00.292541 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286882    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:00.292541 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286884    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:00.292541 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286886    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:00.292541 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286889    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:00.292541 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286892    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:00.292541 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286894    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:00.292541 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286905    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:00.292541 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286908    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:00.292541 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286910    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:00.292541 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286912    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:00.292541 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286915    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:00.293028 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286917    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:00.293028 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286920    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:00.293028 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286922    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:00.293028 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286924    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:00.293028 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286927    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:00.293028 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286929    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:00.293028 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286932    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:00.293028 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286934    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:00.293028 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286936    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:00.293028 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286938    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:00.293028 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286948    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:00.293028 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286951    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:00.293028 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286953    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:00.293028 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286956    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:00.293028 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286958    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:00.293028 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286961    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:00.293028 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286963    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:00.293028 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286965    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:00.293028 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286968    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:00.293481 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286970    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:00.293481 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286972    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:00.293481 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286975    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:00.293481 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286977    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:00.293481 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286979    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:00.293481 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286982    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:00.293481 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286984    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:00.293481 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286987    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:00.293481 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286989    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:00.293481 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.286999    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:00.293481 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287004    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:00.293481 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287007    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:00.293481 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287009    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:00.293481 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287012    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:00.293481 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287015    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:00.293481 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287017    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:00.293481 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287019    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:00.293481 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287022    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:00.293481 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287024    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:00.293906 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287027    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:00.293906 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287029    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:00.293906 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287032    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:00.293906 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287034    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:00.293906 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287036    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:00.293906 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287039    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:00.293906 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287041    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:00.293906 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287043    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:00.293906 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287047    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:00.293906 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287050    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:00.293906 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287053    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:00.293906 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287055    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:00.293906 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287064    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:00.293906 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287067    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:00.293906 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287069    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:00.293906 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287072    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:00.293906 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287074    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:00.293906 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287076    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:00.293906 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287079    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:00.293906 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287081    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:00.294377 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287084    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:00.294377 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287086    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:00.294377 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287088    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:00.294377 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287096    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:00.294377 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287099    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:00.294377 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287101    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:00.294377 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287104    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:00.294377 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287106    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:00.294377 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287108    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:00.294377 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287111    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:00.294377 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287113    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:00.294377 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287115    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:00.294377 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287118    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:00.294377 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287120    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:00.294377 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287122    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:00.294377 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287125    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:00.294377 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287127    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:00.294377 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287129    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:00.294377 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287131    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:00.294797 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287134    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:00.294797 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287136    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:00.294797 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287138    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:00.294797 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.287160    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:19:00.294797 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287195    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:00.294797 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287199    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:00.294797 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287202    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:00.294797 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287204    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:00.294797 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287206    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:00.294797 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287209    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:00.294797 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287211    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:00.294797 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287214    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:00.294797 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287216    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:00.294797 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287218    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:00.294797 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287220    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:00.294797 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287223    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:00.295165 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287225    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:00.295165 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287233    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:00.295165 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287235    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:00.295165 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287240    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:00.295165 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287242    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:00.295165 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287244    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:00.295165 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287247    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:00.295165 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287249    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:00.295165 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287251    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:00.295165 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287254    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:00.295165 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287256    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:00.295165 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287258    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:00.295165 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287261    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:00.295165 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287263    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:00.295165 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287266    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:00.295165 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287269    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:00.295165 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287271    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:00.295165 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287273    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:00.295165 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287276    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:00.295165 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287278    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:00.295931 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287280    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:00.295931 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287282    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:00.295931 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287285    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:00.295931 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287287    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:00.295931 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287289    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:00.295931 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287293    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:00.295931 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287296    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:00.295931 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287299    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:00.295931 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287302    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:00.295931 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287305    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:00.295931 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287307    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:00.295931 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287310    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:00.295931 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287312    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:00.295931 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287315    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:00.295931 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287339    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:00.295931 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287344    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:00.295931 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287346    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:00.295931 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287348    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:00.295931 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287351    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:00.296549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287353    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:00.296549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287356    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:00.296549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287359    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:00.296549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287361    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:00.296549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287364    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:00.296549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287368    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:00.296549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287371    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:00.296549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287373    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:00.296549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287376    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:00.296549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287378    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:00.296549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287380    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:00.296549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287383    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:00.296549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287385    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:00.296549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287387    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:00.296549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287390    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:00.296549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287392    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:00.296549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287394    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:00.296549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287397    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:00.296549 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287399    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:00.296978 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287402    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:00.296978 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287404    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:00.296978 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287406    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:00.296978 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287408    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:00.296978 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287411    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:00.296978 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287413    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:00.296978 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287415    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:00.296978 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287417    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:00.296978 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287420    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:00.296978 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287429    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:00.296978 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287432    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:00.296978 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287434    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:00.296978 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287436    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:00.296978 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287439    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:00.296978 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287441    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:00.296978 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287444    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:00.296978 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287446    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:00.296978 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287448    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:00.296978 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287451    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:00.297433 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287453    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:00.297433 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287455    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:00.297433 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287458    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:00.297433 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287460    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:00.297433 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287462    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:00.297433 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287465    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:00.297433 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287467    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:00.297433 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287469    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:00.297433 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287472    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:00.297433 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287475    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:00.297433 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287477    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:00.297433 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287479    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:00.297433 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287482    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:00.297433 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287484    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:00.297433 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287486    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:00.297433 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287488    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:00.297433 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:00.287491    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:00.297433 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.287544    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 20:19:00.297433 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.289958    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 20:19:00.297433 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.290926    2577 server.go:1014] "Starting client certificate rotation"
Oct 30 20:19:00.297867 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.291027    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:19:00.297867 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.291062    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:19:00.297867 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.293089    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:19:00.298722 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.298700    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:19:00.313232 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.313210    2577 log.go:25] "Validated CRI v1 runtime API"
Oct 30 20:19:00.317034 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.317021    2577 log.go:25] "Validated CRI v1 image API"
Oct 30 20:19:00.317853 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.317830    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 20:19:00.322815 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.322795    2577 fs.go:135] Filesystem UUIDs: map[4068e041-80b4-4ea7-97b6-278deec10bd5:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9286d060-8a15-4e32-8290-6a7f0b52d327:/dev/nvme0n1p3]
Oct 30 20:19:00.322863 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.322815    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 20:19:00.327886 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.327777    2577 manager.go:217] Machine: {Timestamp:2026-10-30 20:19:00.326413301 +0000 UTC m=+0.335761594 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3197392 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27fc0f28adc583cdf1717034fa200d SystemUUID:ec27fc0f-28ad-c583-cdf1-717034fa200d BootID:91fb4842-1ec5-4166-85a0-670e308d1989 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3f:91:8a:a2:c5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3f:91:8a:a2:c5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:e6:88:22:52:60 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 20:19:00.327886 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.327886    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 20:19:00.328005 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.327965    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 20:19:00.330654 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.330634    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 20:19:00.330788 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.330656    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-34.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 20:19:00.330833 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.330790    2577 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 20:19:00.330833 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.330797    2577 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 20:19:00.330833 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.330831    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:19:00.330906 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.330849    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:19:00.331939 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.331928    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 20:19:00.336438 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.336426    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 20:19:00.336548 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.336540    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 20:19:00.337336 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.337315    2577 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 20:19:00.337366 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.337341    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 20:19:00.337366 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.337356    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 20:19:00.337428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.337368    2577 kubelet.go:405] "Adding apiserver pod source"
Oct 30 20:19:00.337428 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.337378    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 20:19:00.337759 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.337747    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:19:00.340038 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.340016    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 20:19:00.347123 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.347110    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:19:00.348072 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.348061    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 20:19:00.348117 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.348097    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 20:19:00.348245 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.348236    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 20:19:00.348276 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.348252    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 20:19:00.348276 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.348258    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 20:19:00.348276 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.348263    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 20:19:00.348276 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.348269    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 20:19:00.348276 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.348275    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 20:19:00.348975 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.348966    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 20:19:00.349008 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.348976    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 20:19:00.349008 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.348983    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 20:19:00.349008 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.348989    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 20:19:00.349008 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.348999    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 20:19:00.349008 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.349007    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 20:19:00.350393 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.350384    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 20:19:00.350425 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.350394    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 20:19:00.354273 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.354261    2577 server.go:1263] "Started kubelet"
Oct 30 20:19:00.354382 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.354317    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 20:19:00.354433 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.354383    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 20:19:00.354489 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.354451    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 20:19:00.354988 ip-10-0-138-34 systemd[1]: Started Kubernetes Kubelet.
Oct 30 20:19:00.358173 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.358148    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 20:19:00.359086 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.359068    2577 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 20:19:00.361375 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:00.361355    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 20:19:00.365096 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.365073    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 20:19:00.365202 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.365107    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 20:19:00.365202 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:00.365171    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-34.us-west-1.compute.internal\" not found"
Oct 30 20:19:00.366088 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.366071    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 20:19:00.366088 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.366077    2577 factory.go:55] Registering systemd factory
Oct 30 20:19:00.366203 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.366102    2577 factory.go:223] Registration of the systemd container factory successfully
Oct 30 20:19:00.366203 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.366152    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 20:19:00.366203 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.366164    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 20:19:00.366351 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.366241    2577 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 20:19:00.366351 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.366250    2577 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 20:19:00.366437 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.366420    2577 factory.go:153] Registering CRI-O factory
Oct 30 20:19:00.366437 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.366435    2577 factory.go:223] Registration of the crio container factory successfully
Oct 30 20:19:00.366506 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.366489    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 20:19:00.366676 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.366587    2577 factory.go:103] Registering Raw factory
Oct 30 20:19:00.366733 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.366688    2577 manager.go:1199] Started watching for new ooms in manager
Oct 30 20:19:00.367770 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.367752    2577 manager.go:319] Starting recovery of all containers
Oct 30 20:19:00.374244 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.374195    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 20:19:00.378890 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.378874    2577 manager.go:324] Recovery completed
Oct 30 20:19:00.383083 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.383066    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:00.384522 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.384498    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:00.384594 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.384539    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:00.384594 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.384555    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:00.385106 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.385093    2577 cpu_manager.go:226] "Starting" policy="none"
Oct 30 20:19:00.385160 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.385103    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 20:19:00.385160 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.385097    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:00.385160 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.385128    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 20:19:00.385748 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.385734    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:00.385801 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.385761    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:00.385801 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.385772    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:00.387258 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.387247    2577 policy_none.go:50] "Start"
Oct 30 20:19:00.387296 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.387262    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 20:19:00.387296 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.387272    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 20:19:00.389259 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.389250    2577 policy_none.go:44] "Start"
Oct 30 20:19:00.423695 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.423649    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 20:19:00.448312 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.446453    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 20:19:00.448312 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.446615    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 20:19:00.448312 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.446640    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 20:19:00.448312 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.446683    2577 manager.go:341] "Starting Device Plugin manager"
Oct 30 20:19:00.448312 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:00.446730    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 20:19:00.448312 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.446740    2577 server.go:87] "Starting device plugin registration server"
Oct 30 20:19:00.448312 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:00.446677    2577 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 20:19:00.448312 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.446998    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 20:19:00.448312 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.447009    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 20:19:00.448312 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.447152    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 20:19:00.448312 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.447215    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 20:19:00.448312 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.447220    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 20:19:00.448312 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:00.447830    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 20:19:00.448312 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:00.447879    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-34.us-west-1.compute.internal\" not found"
Oct 30 20:19:00.495080 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:00.495040    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-34.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 20:19:00.495080 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.495049    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-34.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 20:19:00.496151 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:00.495047    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-34.us-west-1.compute.internal.18e3681aa9fe8178  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-34.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-34.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 20:19:00.354056568 +0000 UTC m=+0.363404843,LastTimestamp:2026-10-30 20:19:00.354056568 +0000 UTC m=+0.363404843,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-34.us-west-1.compute.internal,}"
Oct 30 20:19:00.547277 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.547214    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:00.547432 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.547273    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal"]
Oct 30 20:19:00.547432 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.547357    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:00.548223 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.548205    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:00.548348 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.548242    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:00.548348 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.548260    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:00.548348 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.548258    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:00.548348 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.548297    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:00.548348 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.548302    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:00.548348 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.548316    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:00.550463 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.550449    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:00.550605 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.550591    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:00.550667 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.550628    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:00.551138 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.551122    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:00.551217 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.551140    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:00.551217 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.551164    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:00.551217 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.551174    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:00.551217 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.551148    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:00.551353 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.551225    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:00.553166 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.553152    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:00.553264 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.553176    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:00.553858 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.553842    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:00.553957 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.553867    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:00.553957 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.553878    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:00.556760 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:00.556532    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-34.us-west-1.compute.internal.18e3681aabcf5c6a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-34.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-34.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-34.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 20:19:00.384521322 +0000 UTC m=+0.393869598,LastTimestamp:2026-10-30 20:19:00.384521322 +0000 UTC m=+0.393869598,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-34.us-west-1.compute.internal,}"
Oct 30 20:19:00.557424 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:00.557399    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 20:19:00.557424 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:00.557403    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-34.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 20:19:00.557572 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:00.557402    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 20:19:00.576594 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:00.576571    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-34.us-west-1.compute.internal\" not found" node="ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:00.579196 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:00.579179    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-34.us-west-1.compute.internal\" not found" node="ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:00.580677 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.580663    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 20:19:00.580743 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.580718    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-clr75"
Oct 30 20:19:00.652590 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.652563    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:19:00.667263 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.667239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be68dc42ad591cdc0bc8a690a67f92fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal\" (UID: \"be68dc42ad591cdc0bc8a690a67f92fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:00.667379 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.667265    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be68dc42ad591cdc0bc8a690a67f92fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal\" (UID: \"be68dc42ad591cdc0bc8a690a67f92fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:00.667379 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.667283    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d10e19b03ae8d13f3ef8ccaa1c90976f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal\" (UID: \"d10e19b03ae8d13f3ef8ccaa1c90976f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:00.671807 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.671791    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:00.671847 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:00.671813    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-34.us-west-1.compute.internal\": node \"ip-10-0-138-34.us-west-1.compute.internal\" not found"
Oct 30 20:19:00.768449 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.768422    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d10e19b03ae8d13f3ef8ccaa1c90976f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal\" (UID: \"d10e19b03ae8d13f3ef8ccaa1c90976f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:00.768449 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.768435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d10e19b03ae8d13f3ef8ccaa1c90976f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal\" (UID: \"d10e19b03ae8d13f3ef8ccaa1c90976f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:00.768636 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.768475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be68dc42ad591cdc0bc8a690a67f92fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal\" (UID: \"be68dc42ad591cdc0bc8a690a67f92fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:00.768636 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.768520    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be68dc42ad591cdc0bc8a690a67f92fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal\" (UID: \"be68dc42ad591cdc0bc8a690a67f92fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:00.768636 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.768544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be68dc42ad591cdc0bc8a690a67f92fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal\" (UID: \"be68dc42ad591cdc0bc8a690a67f92fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:00.768636 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.768573    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be68dc42ad591cdc0bc8a690a67f92fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal\" (UID: \"be68dc42ad591cdc0bc8a690a67f92fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:00.808486 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:00.808417    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-34.us-west-1.compute.internal\" not found"
Oct 30 20:19:00.880523 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.880492    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:00.884193 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:00.884172    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:00.909461 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:00.909437    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-34.us-west-1.compute.internal\" not found"
Oct 30 20:19:01.009819 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:01.009772    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-34.us-west-1.compute.internal\" not found"
Oct 30 20:19:01.110446 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:01.110372    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-34.us-west-1.compute.internal\" not found"
Oct 30 20:19:01.210833 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:01.210789    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-34.us-west-1.compute.internal\" not found"
Oct 30 20:19:01.291029 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:01.290991    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 20:19:01.291510 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:01.291151    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 20:19:01.291510 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:01.291207    2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ab4a4e8a057d64e13b2573a04657ce20-dcfeed6681acbb51.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-138-34.us-west-1.compute.internal.18e3681aabcf5c6a\": read tcp 10.0.138.34:51450->77.112.75.128:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-138-34.us-west-1.compute.internal.18e3681aabcf5c6a  default   5041 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-34.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-34.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-34.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 20:19:00 +0000 UTC,LastTimestamp:2026-10-30 20:19:00.548292283 +0000 UTC m=+0.557640558,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-34.us-west-1.compute.internal,}"
Oct 30 20:19:01.311256 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:01.311230    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-34.us-west-1.compute.internal\" not found"
Oct 30 20:19:01.411836 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:01.411766    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-34.us-west-1.compute.internal\" not found"
Oct 30 20:19:01.512059 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:01.512016    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-34.us-west-1.compute.internal\" not found"
Oct 30 20:19:01.531625 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:01.531599    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:19:01.566002 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:01.565977    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:01.588522 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:01.588489    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:14:01 +0000 UTC" deadline="2028-08-06 06:55:58.357772492 +0000 UTC"
Oct 30 20:19:01.588522 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:01.588518    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15490h36m56.769257163s"
Oct 30 20:19:01.629035 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:01.629012    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:19:01.630855 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:01.630841    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:01.688648 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:01.688563    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:19:01.689461 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:01.689445    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:19:01.729515 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:01.729484    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:19:01.965562 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:01.965532    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe68dc42ad591cdc0bc8a690a67f92fa.slice/crio-c3ffd5d5a254a8250cac90d26fab7f14eb6aa66e8814427d6eb8aedd0861b0f0 WatchSource:0}: Error finding container c3ffd5d5a254a8250cac90d26fab7f14eb6aa66e8814427d6eb8aedd0861b0f0: Status 404 returned error can't find the container with id c3ffd5d5a254a8250cac90d26fab7f14eb6aa66e8814427d6eb8aedd0861b0f0
Oct 30 20:19:01.966262 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:01.966234    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10e19b03ae8d13f3ef8ccaa1c90976f.slice/crio-8dc8b359935965bd2c5b6908b383663e73f5b9af88074a89ceda2b99960f54ec WatchSource:0}: Error finding container 8dc8b359935965bd2c5b6908b383663e73f5b9af88074a89ceda2b99960f54ec: Status 404 returned error can't find the container with id 8dc8b359935965bd2c5b6908b383663e73f5b9af88074a89ceda2b99960f54ec
Oct 30 20:19:01.971723 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:01.971709    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:19:02.212163 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.212086    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:19:02.338789 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.338759    2577 apiserver.go:52] "Watching apiserver"
Oct 30 20:19:02.367077 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.367047    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 20:19:02.452155 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.452128    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 20:19:02.452492 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.452444    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal" event={"ID":"be68dc42ad591cdc0bc8a690a67f92fa","Type":"ContainerStarted","Data":"c3ffd5d5a254a8250cac90d26fab7f14eb6aa66e8814427d6eb8aedd0861b0f0"}
Oct 30 20:19:02.452607 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.452519    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-r7mrf","openshift-cluster-node-tuning-operator/tuned-cwm29","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal","openshift-ovn-kubernetes/ovnkube-node-frx59","kube-system/konnectivity-agent-qxwhp","kube-system/kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg","openshift-network-diagnostics/network-check-target-d9xr4","openshift-dns/node-resolver-jbb5l","openshift-multus/multus-additional-cni-plugins-2tgdb","openshift-multus/network-metrics-daemon-vkj2j","kube-system/global-pull-secret-syncer-jkz4p","openshift-image-registry/node-ca-n8gtv","openshift-multus/multus-psqxp"]
Oct 30 20:19:02.456701 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.456683    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.458893 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.458875    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n8gtv"
Oct 30 20:19:02.461262 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.461248    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.463690 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.463673    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.466036 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.466020    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qxwhp"
Oct 30 20:19:02.468818 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.468800    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:02.468924 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:02.468866    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xr4" podUID="b9a64b7e-6436-4415-86f9-735350f24e35"
Oct 30 20:19:02.471302 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.471284    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jbb5l"
Oct 30 20:19:02.473978 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.473962    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r7mrf"
Oct 30 20:19:02.476526 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.476507    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.479624 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.479601    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.480139 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480109    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-var-lib-cni-multus\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.480237 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480155    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-run-multus-certs\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.480237 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480189    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-run-netns\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.480237 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480220    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-run-ovn-kubernetes\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.480383 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb857965-3140-4b4e-982c-af2680bbeff3-ovn-node-metrics-cert\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.480383 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-registration-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.480383 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f82dd16e-0e8a-40ef-87bd-84560a6b9b71-serviceca\") pod \"node-ca-n8gtv\" (UID: \"f82dd16e-0e8a-40ef-87bd-84560a6b9b71\") " pod="openshift-image-registry/node-ca-n8gtv"
Oct 30 20:19:02.480383 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480341    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-cnibin\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.480383 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-var-lib-cni-bin\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.480629 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-hostroot\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.480629 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-etc-openvswitch\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.480629 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480538    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-node-log\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.480629 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb857965-3140-4b4e-982c-af2680bbeff3-ovnkube-config\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.480629 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480606    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1089345a-f93e-4c78-ad1c-ac38961bdf10-agent-certs\") pod \"konnectivity-agent-qxwhp\" (UID: \"1089345a-f93e-4c78-ad1c-ac38961bdf10\") " pod="kube-system/konnectivity-agent-qxwhp"
Oct 30 20:19:02.480843 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480641    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-device-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.480843 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480671    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-kubelet\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.480843 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-run-openvswitch\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.480843 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-run-ovn\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.480843 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-cni-bin\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.480843 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480787    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-socket-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.480843 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kknl7\" (UniqueName: \"kubernetes.io/projected/3e789fe9-f262-4147-b4ee-ec42835f4ed0-kube-api-access-kknl7\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.481158 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f82dd16e-0e8a-40ef-87bd-84560a6b9b71-host\") pod \"node-ca-n8gtv\" (UID: \"f82dd16e-0e8a-40ef-87bd-84560a6b9b71\") " pod="openshift-image-registry/node-ca-n8gtv"
Oct 30 20:19:02.481158 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480868    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtjh9\" (UniqueName: \"kubernetes.io/projected/f82dd16e-0e8a-40ef-87bd-84560a6b9b71-kube-api-access-rtjh9\") pod \"node-ca-n8gtv\" (UID: \"f82dd16e-0e8a-40ef-87bd-84560a6b9b71\") " pod="openshift-image-registry/node-ca-n8gtv"
Oct 30 20:19:02.481158 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480898    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-var-lib-kubelet\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.481158 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-daemon-config\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.481158 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480959    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-slash\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.481158 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.480981    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-run-systemd\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.481158 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.481010    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-log-socket\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.481158 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.481038    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-cni-netd\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.481158 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.481069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-sys-fs\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.481158 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.481097    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-socket-dir-parent\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.481158 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.481122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-run-k8s-cni-cncf-io\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.481158 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.481150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-run-netns\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.481810 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.481177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-systemd-units\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.481810 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.481210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.481810 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.481239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb857965-3140-4b4e-982c-af2680bbeff3-env-overrides\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.481810 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.481269    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.481810 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.481304    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-conf-dir\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.481810 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.481327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-etc-kubernetes\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.481810 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.481357    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-system-cni-dir\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.481810 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.481384    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-cni-dir\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.481810 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.481433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a0b6cf3-8303-4d98-b22f-f8a313721702-cni-binary-copy\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.481810 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.481459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-var-lib-openvswitch\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.481810 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.481488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqljx\" (UniqueName: \"kubernetes.io/projected/fb857965-3140-4b4e-982c-af2680bbeff3-kube-api-access-nqljx\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.481810 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.481518    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1089345a-f93e-4c78-ad1c-ac38961bdf10-konnectivity-ca\") pod \"konnectivity-agent-qxwhp\" (UID: \"1089345a-f93e-4c78-ad1c-ac38961bdf10\") " pod="kube-system/konnectivity-agent-qxwhp"
Oct 30 20:19:02.481810 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.481550    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl5jc\" (UniqueName: \"kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc\") pod \"network-check-target-d9xr4\" (UID: \"b9a64b7e-6436-4415-86f9-735350f24e35\") " pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:02.481810 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.481580    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-etc-selinux\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.481810 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.481615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxckk\" (UniqueName: \"kubernetes.io/projected/0a0b6cf3-8303-4d98-b22f-f8a313721702-kube-api-access-fxckk\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.481810 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.481645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb857965-3140-4b4e-982c-af2680bbeff3-ovnkube-script-lib\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.482891 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.482867    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:02.482996 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:02.482948    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkj2j" podUID="c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256"
Oct 30 20:19:02.485843 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.485823    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal" event={"ID":"d10e19b03ae8d13f3ef8ccaa1c90976f","Type":"ContainerStarted","Data":"8dc8b359935965bd2c5b6908b383663e73f5b9af88074a89ceda2b99960f54ec"}
Oct 30 20:19:02.485932 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.485882    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:02.485932 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:02.485926    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkz4p" podUID="f399aaa3-b634-40e9-9ca7-0941f0ec2925"
Oct 30 20:19:02.514627 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.514607    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 20:19:02.514778 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.514760    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 20:19:02.514838 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.514790    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lm5vf\""
Oct 30 20:19:02.514838 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.514815    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 20:19:02.516033 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.516019    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 20:19:02.516365 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.516349    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 20:19:02.516365 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.516360    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 20:19:02.516489 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.516385    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kv88f\""
Oct 30 20:19:02.517996 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.517966    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 20:19:02.518060 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.518048    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 20:19:02.518185 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.518174    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 20:19:02.518218 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.518210    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 20:19:02.518275 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.518264    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-k74qc\""
Oct 30 20:19:02.520507 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.520494    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 20:19:02.520630 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.520615    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5x4ch\""
Oct 30 20:19:02.520805 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.520791    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 20:19:02.567538 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.567516    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 20:19:02.569813 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.569798    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 20:19:02.582348 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6s8r\" (UniqueName: \"kubernetes.io/projected/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-kube-api-access-s6s8r\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.582408 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582355    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-run-k8s-cni-cncf-io\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.582408 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582395    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.582472 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582450    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.582472 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582462    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-run-k8s-cni-cncf-io\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.582536 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgw7z\" (UniqueName: \"kubernetes.io/projected/c85a95c6-4b03-47db-a7a3-f702a80fe01f-kube-api-access-tgw7z\") pod \"iptables-alerter-r7mrf\" (UID: \"c85a95c6-4b03-47db-a7a3-f702a80fe01f\") " pod="openshift-network-operator/iptables-alerter-r7mrf"
Oct 30 20:19:02.582536 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582503    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51727502-b532-46d0-828c-140b8eda2ba3-cni-binary-copy\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.582633 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582544    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51727502-b532-46d0-828c-140b8eda2ba3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.582633 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-sys\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.582633 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582588    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-conf-dir\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.582782 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c85a95c6-4b03-47db-a7a3-f702a80fe01f-iptables-alerter-script\") pod \"iptables-alerter-r7mrf\" (UID: \"c85a95c6-4b03-47db-a7a3-f702a80fe01f\") " pod="openshift-network-operator/iptables-alerter-r7mrf"
Oct 30 20:19:02.582782 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-sysctl-conf\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.582782 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-lib-modules\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.582782 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582714    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-conf-dir\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.582782 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582763    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-system-cni-dir\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.583024 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582805    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-cni-dir\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.583024 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a0b6cf3-8303-4d98-b22f-f8a313721702-cni-binary-copy\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.583024 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582854    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-var-lib-openvswitch\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.583024 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582874    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-system-cni-dir\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.583024 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqljx\" (UniqueName: \"kubernetes.io/projected/fb857965-3140-4b4e-982c-af2680bbeff3-kube-api-access-nqljx\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.583024 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582905    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-var-lib-openvswitch\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.583024 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582909    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl5jc\" (UniqueName: \"kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc\") pod \"network-check-target-d9xr4\" (UID: \"b9a64b7e-6436-4415-86f9-735350f24e35\") " pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:02.583024 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f399aaa3-b634-40e9-9ca7-0941f0ec2925-kubelet-config\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:02.583024 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582962    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f399aaa3-b634-40e9-9ca7-0941f0ec2925-dbus\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:02.583024 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.582983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxckk\" (UniqueName: \"kubernetes.io/projected/0a0b6cf3-8303-4d98-b22f-f8a313721702-kube-api-access-fxckk\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.583024 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb857965-3140-4b4e-982c-af2680bbeff3-ovnkube-script-lib\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.583505 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583037    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6tv5\" (UniqueName: \"kubernetes.io/projected/51727502-b532-46d0-828c-140b8eda2ba3-kube-api-access-s6tv5\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.583505 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583052    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-cni-dir\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.583505 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c2c9\" (UniqueName: \"kubernetes.io/projected/bd71c9e1-948b-41cc-9d05-a65a556c8ea7-kube-api-access-4c2c9\") pod \"node-resolver-jbb5l\" (UID: \"bd71c9e1-948b-41cc-9d05-a65a556c8ea7\") " pod="openshift-dns/node-resolver-jbb5l"
Oct 30 20:19:02.583505 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583083    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-tmp\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.583505 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583188    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-var-lib-cni-multus\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.583505 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583216    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-var-lib-cni-multus\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.583505 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-run-multus-certs\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.583505 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583247    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-run-multus-certs\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.583505 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583268    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-run-ovn-kubernetes\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.583505 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb857965-3140-4b4e-982c-af2680bbeff3-ovn-node-metrics-cert\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.583505 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583311    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-registration-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.583505 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583355    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f82dd16e-0e8a-40ef-87bd-84560a6b9b71-serviceca\") pod \"node-ca-n8gtv\" (UID: \"f82dd16e-0e8a-40ef-87bd-84560a6b9b71\") " pod="openshift-image-registry/node-ca-n8gtv"
Oct 30 20:19:02.583505 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583372    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-run-ovn-kubernetes\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.583505 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583383    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bd71c9e1-948b-41cc-9d05-a65a556c8ea7-hosts-file\") pod \"node-resolver-jbb5l\" (UID: \"bd71c9e1-948b-41cc-9d05-a65a556c8ea7\") " pod="openshift-dns/node-resolver-jbb5l"
Oct 30 20:19:02.583505 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-host\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.583505 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583429    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-registration-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.583505 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583450    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-var-lib-cni-bin\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.584279 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-etc-openvswitch\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.584279 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-node-log\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.584279 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583518    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-var-lib-cni-bin\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.584279 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583554    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-node-log\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.584279 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583540    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-etc-openvswitch\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.584279 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb857965-3140-4b4e-982c-af2680bbeff3-ovnkube-config\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.584279 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1089345a-f93e-4c78-ad1c-ac38961bdf10-agent-certs\") pod \"konnectivity-agent-qxwhp\" (UID: \"1089345a-f93e-4c78-ad1c-ac38961bdf10\") " pod="kube-system/konnectivity-agent-qxwhp"
Oct 30 20:19:02.584279 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583611    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a0b6cf3-8303-4d98-b22f-f8a313721702-cni-binary-copy\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.584279 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f82dd16e-0e8a-40ef-87bd-84560a6b9b71-host\") pod \"node-ca-n8gtv\" (UID: \"f82dd16e-0e8a-40ef-87bd-84560a6b9b71\") " pod="openshift-image-registry/node-ca-n8gtv"
Oct 30 20:19:02.584279 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583653    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb857965-3140-4b4e-982c-af2680bbeff3-ovnkube-script-lib\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.584279 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583675    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f82dd16e-0e8a-40ef-87bd-84560a6b9b71-host\") pod \"node-ca-n8gtv\" (UID: \"f82dd16e-0e8a-40ef-87bd-84560a6b9b71\") " pod="openshift-image-registry/node-ca-n8gtv"
Oct 30 20:19:02.584279 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51727502-b532-46d0-828c-140b8eda2ba3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.584279 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583712    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51727502-b532-46d0-828c-140b8eda2ba3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.584279 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583731    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-kubelet\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.584279 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-socket-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.584279 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583769    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-kubelet\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.584279 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583796    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:02.584956 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583813    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-modprobe-d\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.584956 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-sysconfig\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.584956 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583830    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f82dd16e-0e8a-40ef-87bd-84560a6b9b71-serviceca\") pod \"node-ca-n8gtv\" (UID: \"f82dd16e-0e8a-40ef-87bd-84560a6b9b71\") " pod="openshift-image-registry/node-ca-n8gtv"
Oct 30 20:19:02.584956 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-var-lib-kubelet\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.584956 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-var-lib-kubelet\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.584956 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-daemon-config\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.584956 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583937    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-socket-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.584956 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583941    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-var-lib-kubelet\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.584956 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-slash\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.584956 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.583990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-log-socket\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.584956 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584007    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-socket-dir-parent\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.584956 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584023    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-run-netns\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.584956 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584028    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-slash\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.584956 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584052    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-log-socket\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.584956 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584066    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-run-netns\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.584956 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584077    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-systemd-units\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.584956 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584081    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-socket-dir-parent\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.585466 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584116    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.585466 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584124    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-systemd-units\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.585466 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb857965-3140-4b4e-982c-af2680bbeff3-env-overrides\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.585466 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-etc-kubernetes\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.585466 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584186    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.585466 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584197    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs\") pod \"network-metrics-daemon-vkj2j\" (UID: \"c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256\") " pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:02.585466 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584234    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-etc-kubernetes\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.585466 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584242    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq5ft\" (UniqueName: \"kubernetes.io/projected/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-kube-api-access-vq5ft\") pod \"network-metrics-daemon-vkj2j\" (UID: \"c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256\") " pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:02.585466 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584263    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd71c9e1-948b-41cc-9d05-a65a556c8ea7-tmp-dir\") pod \"node-resolver-jbb5l\" (UID: \"bd71c9e1-948b-41cc-9d05-a65a556c8ea7\") " pod="openshift-dns/node-resolver-jbb5l"
Oct 30 20:19:02.585466 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1089345a-f93e-4c78-ad1c-ac38961bdf10-konnectivity-ca\") pod \"konnectivity-agent-qxwhp\" (UID: \"1089345a-f93e-4c78-ad1c-ac38961bdf10\") " pod="kube-system/konnectivity-agent-qxwhp"
Oct 30 20:19:02.585466 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584300    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-etc-selinux\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.585466 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51727502-b532-46d0-828c-140b8eda2ba3-cnibin\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.585466 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584347    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-daemon-config\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.585466 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584386    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-kubernetes\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.585466 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584405    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c85a95c6-4b03-47db-a7a3-f702a80fe01f-host-slash\") pod \"iptables-alerter-r7mrf\" (UID: \"c85a95c6-4b03-47db-a7a3-f702a80fe01f\") " pod="openshift-network-operator/iptables-alerter-r7mrf"
Oct 30 20:19:02.585466 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-run-netns\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.585466 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584422    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-etc-selinux\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.586876 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584459    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-run-netns\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.586876 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584471    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-cnibin\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.586876 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584491    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-hostroot\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.586876 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584494    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb857965-3140-4b4e-982c-af2680bbeff3-ovnkube-config\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.586876 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584506    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-device-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.586876 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584512    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-cnibin\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.586876 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kknl7\" (UniqueName: \"kubernetes.io/projected/3e789fe9-f262-4147-b4ee-ec42835f4ed0-kube-api-access-kknl7\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.586876 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584535    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-hostroot\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.586876 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-run-openvswitch\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.586876 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584563    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-device-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.586876 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584583    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-run-openvswitch\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.586876 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-run-ovn\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.586876 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584595    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-run-ovn\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.586876 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-cni-bin\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.586876 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtjh9\" (UniqueName: \"kubernetes.io/projected/f82dd16e-0e8a-40ef-87bd-84560a6b9b71-kube-api-access-rtjh9\") pod \"node-ca-n8gtv\" (UID: \"f82dd16e-0e8a-40ef-87bd-84560a6b9b71\") " pod="openshift-image-registry/node-ca-n8gtv"
Oct 30 20:19:02.586876 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584644    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51727502-b532-46d0-828c-140b8eda2ba3-system-cni-dir\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.586876 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584660    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-sysctl-d\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.586876 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584674    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-run\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.587370 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-run-systemd\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.587370 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-cni-netd\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.587370 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-sys-fs\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.587370 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584721    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-run-systemd\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.587370 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-systemd\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.587370 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584750    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-tuned\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.587370 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584754    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-cni-netd\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.587370 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584776    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-cni-bin\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.587370 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.584814    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-sys-fs\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.618364 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.618340    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 20:19:02.618458 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.618373    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lcz2c"
Oct 30 20:19:02.623938 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.623918    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 20:19:02.624066 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.623976    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 20:19:02.624066 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.623978    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 20:19:02.624066 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.624004    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vn2fz\""
Oct 30 20:19:02.624294 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.624114    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 20:19:02.624294 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.624125    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 20:19:02.624294 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.624137    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:19:02.624294 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.624180    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 20:19:02.624294 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.624185    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 20:19:02.624294 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.624182    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 20:19:02.624599 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.624320    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 20:19:02.624599 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.624339    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 20:19:02.624599 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.624360    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 20:19:02.624599 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.624354    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:19:02.624599 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.624376    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 20:19:02.624599 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.624393    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gbmn9\""
Oct 30 20:19:02.624599 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.624395    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4gwdx\""
Oct 30 20:19:02.624599 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.624421    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2zwff\""
Oct 30 20:19:02.624599 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.624476    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6wxmv\""
Oct 30 20:19:02.624933 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.624729    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb857965-3140-4b4e-982c-af2680bbeff3-env-overrides\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.624933 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.624853    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1089345a-f93e-4c78-ad1c-ac38961bdf10-konnectivity-ca\") pod \"konnectivity-agent-qxwhp\" (UID: \"1089345a-f93e-4c78-ad1c-ac38961bdf10\") " pod="kube-system/konnectivity-agent-qxwhp"
Oct 30 20:19:02.633925 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.633897    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 20:19:02.637251 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.637226    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb857965-3140-4b4e-982c-af2680bbeff3-ovn-node-metrics-cert\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.637348 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.637304    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1089345a-f93e-4c78-ad1c-ac38961bdf10-agent-certs\") pod \"konnectivity-agent-qxwhp\" (UID: \"1089345a-f93e-4c78-ad1c-ac38961bdf10\") " pod="kube-system/konnectivity-agent-qxwhp"
Oct 30 20:19:02.643354 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:02.643319    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:19:02.643354 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:02.643351    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:19:02.643532 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:02.643361    2577 projected.go:196] Error preparing data for projected volume kube-api-access-vl5jc for pod openshift-network-diagnostics/network-check-target-d9xr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:02.643532 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:02.643429    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc podName:b9a64b7e-6436-4415-86f9-735350f24e35 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:03.143406182 +0000 UTC m=+3.152754467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vl5jc" (UniqueName: "kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc") pod "network-check-target-d9xr4" (UID: "b9a64b7e-6436-4415-86f9-735350f24e35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:02.646043 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.646022    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqljx\" (UniqueName: \"kubernetes.io/projected/fb857965-3140-4b4e-982c-af2680bbeff3-kube-api-access-nqljx\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.646138 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.646067    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxckk\" (UniqueName: \"kubernetes.io/projected/0a0b6cf3-8303-4d98-b22f-f8a313721702-kube-api-access-fxckk\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.646204 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.646162    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtjh9\" (UniqueName: \"kubernetes.io/projected/f82dd16e-0e8a-40ef-87bd-84560a6b9b71-kube-api-access-rtjh9\") pod \"node-ca-n8gtv\" (UID: \"f82dd16e-0e8a-40ef-87bd-84560a6b9b71\") " pod="openshift-image-registry/node-ca-n8gtv"
Oct 30 20:19:02.646421 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.646351    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kknl7\" (UniqueName: \"kubernetes.io/projected/3e789fe9-f262-4147-b4ee-ec42835f4ed0-kube-api-access-kknl7\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.685957 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.685936    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-systemd\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.686058 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.685961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-tuned\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.686058 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.685979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6s8r\" (UniqueName: \"kubernetes.io/projected/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-kube-api-access-s6s8r\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.686058 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686000    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgw7z\" (UniqueName: \"kubernetes.io/projected/c85a95c6-4b03-47db-a7a3-f702a80fe01f-kube-api-access-tgw7z\") pod \"iptables-alerter-r7mrf\" (UID: \"c85a95c6-4b03-47db-a7a3-f702a80fe01f\") " pod="openshift-network-operator/iptables-alerter-r7mrf"
Oct 30 20:19:02.686058 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51727502-b532-46d0-828c-140b8eda2ba3-cni-binary-copy\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.686058 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51727502-b532-46d0-828c-140b8eda2ba3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.686058 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686052    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-sys\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.686483 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686062    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-systemd\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.686483 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c85a95c6-4b03-47db-a7a3-f702a80fe01f-iptables-alerter-script\") pod \"iptables-alerter-r7mrf\" (UID: \"c85a95c6-4b03-47db-a7a3-f702a80fe01f\") " pod="openshift-network-operator/iptables-alerter-r7mrf"
Oct 30 20:19:02.686483 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686130    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-sys\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.686483 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-sysctl-conf\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.686483 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-lib-modules\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.686483 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686206    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f399aaa3-b634-40e9-9ca7-0941f0ec2925-kubelet-config\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:02.686483 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686233    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f399aaa3-b634-40e9-9ca7-0941f0ec2925-dbus\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:02.686483 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686290    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f399aaa3-b634-40e9-9ca7-0941f0ec2925-kubelet-config\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:02.686483 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686300    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f399aaa3-b634-40e9-9ca7-0941f0ec2925-dbus\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:02.686483 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686353    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6tv5\" (UniqueName: \"kubernetes.io/projected/51727502-b532-46d0-828c-140b8eda2ba3-kube-api-access-s6tv5\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.686483 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c2c9\" (UniqueName: \"kubernetes.io/projected/bd71c9e1-948b-41cc-9d05-a65a556c8ea7-kube-api-access-4c2c9\") pod \"node-resolver-jbb5l\" (UID: \"bd71c9e1-948b-41cc-9d05-a65a556c8ea7\") " pod="openshift-dns/node-resolver-jbb5l"
Oct 30 20:19:02.686483 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686402    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-lib-modules\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.686483 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686408    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-sysctl-conf\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.686483 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-tmp\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.686483 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bd71c9e1-948b-41cc-9d05-a65a556c8ea7-hosts-file\") pod \"node-resolver-jbb5l\" (UID: \"bd71c9e1-948b-41cc-9d05-a65a556c8ea7\") " pod="openshift-dns/node-resolver-jbb5l"
Oct 30 20:19:02.686483 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-host\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.687128 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51727502-b532-46d0-828c-140b8eda2ba3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.687128 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51727502-b532-46d0-828c-140b8eda2ba3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.687128 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686560    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bd71c9e1-948b-41cc-9d05-a65a556c8ea7-hosts-file\") pod \"node-resolver-jbb5l\" (UID: \"bd71c9e1-948b-41cc-9d05-a65a556c8ea7\") " pod="openshift-dns/node-resolver-jbb5l"
Oct 30 20:19:02.687128 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:02.687128 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-modprobe-d\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.687128 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-sysconfig\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.687128 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686649    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51727502-b532-46d0-828c-140b8eda2ba3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.687128 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-var-lib-kubelet\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.687128 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686693    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-var-lib-kubelet\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.687128 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs\") pod \"network-metrics-daemon-vkj2j\" (UID: \"c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256\") " pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:02.687640 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.687186    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c85a95c6-4b03-47db-a7a3-f702a80fe01f-iptables-alerter-script\") pod \"iptables-alerter-r7mrf\" (UID: \"c85a95c6-4b03-47db-a7a3-f702a80fe01f\") " pod="openshift-network-operator/iptables-alerter-r7mrf"
Oct 30 20:19:02.687640 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.687224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq5ft\" (UniqueName: \"kubernetes.io/projected/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-kube-api-access-vq5ft\") pod \"network-metrics-daemon-vkj2j\" (UID: \"c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256\") " pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:02.687640 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.687283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd71c9e1-948b-41cc-9d05-a65a556c8ea7-tmp-dir\") pod \"node-resolver-jbb5l\" (UID: \"bd71c9e1-948b-41cc-9d05-a65a556c8ea7\") " pod="openshift-dns/node-resolver-jbb5l"
Oct 30 20:19:02.687640 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.687293    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51727502-b532-46d0-828c-140b8eda2ba3-cni-binary-copy\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.687640 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.686723    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51727502-b532-46d0-828c-140b8eda2ba3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.687640 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.687362    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-host\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.687640 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.687374    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-sysconfig\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.687640 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:02.686632    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:02.687640 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:02.687509    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret podName:f399aaa3-b634-40e9-9ca7-0941f0ec2925 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:03.187487207 +0000 UTC m=+3.196835493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret") pod "global-pull-secret-syncer-jkz4p" (UID: "f399aaa3-b634-40e9-9ca7-0941f0ec2925") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:02.687640 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.687546    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-modprobe-d\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.687640 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.687612    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd71c9e1-948b-41cc-9d05-a65a556c8ea7-tmp-dir\") pod \"node-resolver-jbb5l\" (UID: \"bd71c9e1-948b-41cc-9d05-a65a556c8ea7\") " pod="openshift-dns/node-resolver-jbb5l"
Oct 30 20:19:02.688200 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.687682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51727502-b532-46d0-828c-140b8eda2ba3-cnibin\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.688200 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:02.687708    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:02.688200 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.687721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-kubernetes\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.688200 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.687747    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51727502-b532-46d0-828c-140b8eda2ba3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.688200 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.687758    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c85a95c6-4b03-47db-a7a3-f702a80fe01f-host-slash\") pod \"iptables-alerter-r7mrf\" (UID: \"c85a95c6-4b03-47db-a7a3-f702a80fe01f\") " pod="openshift-network-operator/iptables-alerter-r7mrf"
Oct 30 20:19:02.688200 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:02.687769    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs podName:c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:03.187750861 +0000 UTC m=+3.197099130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs") pod "network-metrics-daemon-vkj2j" (UID: "c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:02.688200 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.687802    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c85a95c6-4b03-47db-a7a3-f702a80fe01f-host-slash\") pod \"iptables-alerter-r7mrf\" (UID: \"c85a95c6-4b03-47db-a7a3-f702a80fe01f\") " pod="openshift-network-operator/iptables-alerter-r7mrf"
Oct 30 20:19:02.688200 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.687831    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-kubernetes\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.688200 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.687869    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51727502-b532-46d0-828c-140b8eda2ba3-cnibin\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.688200 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.687960    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51727502-b532-46d0-828c-140b8eda2ba3-system-cni-dir\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.688200 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.688005    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-sysctl-d\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.688200 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.688013    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51727502-b532-46d0-828c-140b8eda2ba3-system-cni-dir\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.688200 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.688032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-run\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.688200 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.688128    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-sysctl-d\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.688200 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.688131    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-run\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.688974 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.688948    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-tmp\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.689039 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.688981    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-tuned\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.748490 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.748420    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6tv5\" (UniqueName: \"kubernetes.io/projected/51727502-b532-46d0-828c-140b8eda2ba3-kube-api-access-s6tv5\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.748650 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.748632    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6s8r\" (UniqueName: \"kubernetes.io/projected/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-kube-api-access-s6s8r\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.748724 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.748672    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgw7z\" (UniqueName: \"kubernetes.io/projected/c85a95c6-4b03-47db-a7a3-f702a80fe01f-kube-api-access-tgw7z\") pod \"iptables-alerter-r7mrf\" (UID: \"c85a95c6-4b03-47db-a7a3-f702a80fe01f\") " pod="openshift-network-operator/iptables-alerter-r7mrf"
Oct 30 20:19:02.748775 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.748743    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c2c9\" (UniqueName: \"kubernetes.io/projected/bd71c9e1-948b-41cc-9d05-a65a556c8ea7-kube-api-access-4c2c9\") pod \"node-resolver-jbb5l\" (UID: \"bd71c9e1-948b-41cc-9d05-a65a556c8ea7\") " pod="openshift-dns/node-resolver-jbb5l"
Oct 30 20:19:02.749436 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.749399    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq5ft\" (UniqueName: \"kubernetes.io/projected/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-kube-api-access-vq5ft\") pod \"network-metrics-daemon-vkj2j\" (UID: \"c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256\") " pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:02.772269 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.772248    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:02.775471 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.775454    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n8gtv"
Oct 30 20:19:02.782823 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.782804    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-psqxp"
Oct 30 20:19:02.783914 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:02.783894    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e789fe9_f262_4147_b4ee_ec42835f4ed0.slice/crio-ade061868534202ccd7cfb197607ca9672e89ef35382f8ddf9c78c1c1a9a3d4c WatchSource:0}: Error finding container ade061868534202ccd7cfb197607ca9672e89ef35382f8ddf9c78c1c1a9a3d4c: Status 404 returned error can't find the container with id ade061868534202ccd7cfb197607ca9672e89ef35382f8ddf9c78c1c1a9a3d4c
Oct 30 20:19:02.784567 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:02.784544    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf82dd16e_0e8a_40ef_87bd_84560a6b9b71.slice/crio-e3988a49523b442fe23502c97df5c0cd28a0c7739f7d5dcae503f258341d806e WatchSource:0}: Error finding container e3988a49523b442fe23502c97df5c0cd28a0c7739f7d5dcae503f258341d806e: Status 404 returned error can't find the container with id e3988a49523b442fe23502c97df5c0cd28a0c7739f7d5dcae503f258341d806e
Oct 30 20:19:02.793265 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:02.793244    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0b6cf3_8303_4d98_b22f_f8a313721702.slice/crio-103e23ae734c5d91614d33935ae0159a9552ae90771906b38eee3ed0993b7e21 WatchSource:0}: Error finding container 103e23ae734c5d91614d33935ae0159a9552ae90771906b38eee3ed0993b7e21: Status 404 returned error can't find the container with id 103e23ae734c5d91614d33935ae0159a9552ae90771906b38eee3ed0993b7e21
Oct 30 20:19:02.798879 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.798861    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qxwhp"
Oct 30 20:19:02.801064 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.801052    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jbb5l"
Oct 30 20:19:02.808028 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.808009    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:02.810137 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.810111    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r7mrf"
Oct 30 20:19:02.811294 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:02.811273    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1089345a_f93e_4c78_ad1c_ac38961bdf10.slice/crio-bc880e8168a5368cb92bfb91773c903408bdbed296f07749b8ad988036d6d243 WatchSource:0}: Error finding container bc880e8168a5368cb92bfb91773c903408bdbed296f07749b8ad988036d6d243: Status 404 returned error can't find the container with id bc880e8168a5368cb92bfb91773c903408bdbed296f07749b8ad988036d6d243
Oct 30 20:19:02.812714 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:02.812650    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd71c9e1_948b_41cc_9d05_a65a556c8ea7.slice/crio-a24d8a5275b2394b165e3211e0c7a89cc761b741205be4feb36b8be702674f75 WatchSource:0}: Error finding container a24d8a5275b2394b165e3211e0c7a89cc761b741205be4feb36b8be702674f75: Status 404 returned error can't find the container with id a24d8a5275b2394b165e3211e0c7a89cc761b741205be4feb36b8be702674f75
Oct 30 20:19:02.812760 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.812716    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:02.818983 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:02.818884    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb857965_3140_4b4e_982c_af2680bbeff3.slice/crio-bc008738dd94a51cd77630beed910b883b0a0d824345d260903c8f370fe0820d WatchSource:0}: Error finding container bc008738dd94a51cd77630beed910b883b0a0d824345d260903c8f370fe0820d: Status 404 returned error can't find the container with id bc008738dd94a51cd77630beed910b883b0a0d824345d260903c8f370fe0820d
Oct 30 20:19:02.822588 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:02.822503    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85a95c6_4b03_47db_a7a3_f702a80fe01f.slice/crio-4e41ee22de1315a865116b5288b8142dfa27eefcb799f1cc10bd405a74216d6b WatchSource:0}: Error finding container 4e41ee22de1315a865116b5288b8142dfa27eefcb799f1cc10bd405a74216d6b: Status 404 returned error can't find the container with id 4e41ee22de1315a865116b5288b8142dfa27eefcb799f1cc10bd405a74216d6b
Oct 30 20:19:02.825369 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:02.825342    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9aa5d5_0be0_4914_a6cc_bd54151feb49.slice/crio-08b2d4755885590e411c540f50204f15a5e58d708f29971226fd9a9303766528 WatchSource:0}: Error finding container 08b2d4755885590e411c540f50204f15a5e58d708f29971226fd9a9303766528: Status 404 returned error can't find the container with id 08b2d4755885590e411c540f50204f15a5e58d708f29971226fd9a9303766528
Oct 30 20:19:02.829992 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:02.829974    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:02.839278 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:02.839256    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51727502_b532_46d0_828c_140b8eda2ba3.slice/crio-162c5e4f2c2fb66c881fa64be8b9da0cf6c2625cd4235006976d33a682598f1e WatchSource:0}: Error finding container 162c5e4f2c2fb66c881fa64be8b9da0cf6c2625cd4235006976d33a682598f1e: Status 404 returned error can't find the container with id 162c5e4f2c2fb66c881fa64be8b9da0cf6c2625cd4235006976d33a682598f1e
Oct 30 20:19:03.191042 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:03.190953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl5jc\" (UniqueName: \"kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc\") pod \"network-check-target-d9xr4\" (UID: \"b9a64b7e-6436-4415-86f9-735350f24e35\") " pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:03.191042 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:03.191010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:03.191240 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:03.191050    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs\") pod \"network-metrics-daemon-vkj2j\" (UID: \"c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256\") " pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:03.191240 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:03.191196    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:03.191310 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:03.191259    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs podName:c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:04.191239332 +0000 UTC m=+4.200587613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs") pod "network-metrics-daemon-vkj2j" (UID: "c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:03.191384 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:03.191344    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:19:03.191384 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:03.191359    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:19:03.191471 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:03.191387    2577 projected.go:196] Error preparing data for projected volume kube-api-access-vl5jc for pod openshift-network-diagnostics/network-check-target-d9xr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:03.191471 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:03.191431    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc podName:b9a64b7e-6436-4415-86f9-735350f24e35 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:04.19141826 +0000 UTC m=+4.200766543 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vl5jc" (UniqueName: "kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc") pod "network-check-target-d9xr4" (UID: "b9a64b7e-6436-4415-86f9-735350f24e35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:03.191550 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:03.191500    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:03.191550 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:03.191534    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret podName:f399aaa3-b634-40e9-9ca7-0941f0ec2925 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:04.191523877 +0000 UTC m=+4.200872169 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret") pod "global-pull-secret-syncer-jkz4p" (UID: "f399aaa3-b634-40e9-9ca7-0941f0ec2925") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:03.458070 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:03.457973    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tgdb" event={"ID":"51727502-b532-46d0-828c-140b8eda2ba3","Type":"ContainerStarted","Data":"162c5e4f2c2fb66c881fa64be8b9da0cf6c2625cd4235006976d33a682598f1e"}
Oct 30 20:19:03.459475 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:03.459435    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cwm29" event={"ID":"5f9aa5d5-0be0-4914-a6cc-bd54151feb49","Type":"ContainerStarted","Data":"08b2d4755885590e411c540f50204f15a5e58d708f29971226fd9a9303766528"}
Oct 30 20:19:03.460782 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:03.460759    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qxwhp" event={"ID":"1089345a-f93e-4c78-ad1c-ac38961bdf10","Type":"ContainerStarted","Data":"bc880e8168a5368cb92bfb91773c903408bdbed296f07749b8ad988036d6d243"}
Oct 30 20:19:03.461929 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:03.461898    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-psqxp" event={"ID":"0a0b6cf3-8303-4d98-b22f-f8a313721702","Type":"ContainerStarted","Data":"103e23ae734c5d91614d33935ae0159a9552ae90771906b38eee3ed0993b7e21"}
Oct 30 20:19:03.463415 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:03.463383    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n8gtv" event={"ID":"f82dd16e-0e8a-40ef-87bd-84560a6b9b71","Type":"ContainerStarted","Data":"e3988a49523b442fe23502c97df5c0cd28a0c7739f7d5dcae503f258341d806e"}
Oct 30 20:19:03.464896 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:03.464871    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r7mrf" event={"ID":"c85a95c6-4b03-47db-a7a3-f702a80fe01f","Type":"ContainerStarted","Data":"4e41ee22de1315a865116b5288b8142dfa27eefcb799f1cc10bd405a74216d6b"}
Oct 30 20:19:03.466406 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:03.466382    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frx59" event={"ID":"fb857965-3140-4b4e-982c-af2680bbeff3","Type":"ContainerStarted","Data":"bc008738dd94a51cd77630beed910b883b0a0d824345d260903c8f370fe0820d"}
Oct 30 20:19:03.467838 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:03.467816    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jbb5l" event={"ID":"bd71c9e1-948b-41cc-9d05-a65a556c8ea7","Type":"ContainerStarted","Data":"a24d8a5275b2394b165e3211e0c7a89cc761b741205be4feb36b8be702674f75"}
Oct 30 20:19:03.469608 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:03.469581    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg" event={"ID":"3e789fe9-f262-4147-b4ee-ec42835f4ed0","Type":"ContainerStarted","Data":"ade061868534202ccd7cfb197607ca9672e89ef35382f8ddf9c78c1c1a9a3d4c"}
Oct 30 20:19:03.623524 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:03.623458    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:14:03 +0000 UTC" deadline="2028-04-12 12:28:36.746576297 +0000 UTC"
Oct 30 20:19:03.623524 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:03.623515    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12712h9m33.123066525s"
Oct 30 20:19:04.201523 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:04.200945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl5jc\" (UniqueName: \"kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc\") pod \"network-check-target-d9xr4\" (UID: \"b9a64b7e-6436-4415-86f9-735350f24e35\") " pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:04.201523 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:04.200996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:04.201523 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:04.201050    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs\") pod \"network-metrics-daemon-vkj2j\" (UID: \"c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256\") " pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:04.201523 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:04.201206    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:04.201523 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:04.201278    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs podName:c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:06.201246769 +0000 UTC m=+6.210595049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs") pod "network-metrics-daemon-vkj2j" (UID: "c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:04.201523 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:04.201441    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:19:04.201523 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:04.201459    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:19:04.201523 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:04.201464    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:04.201523 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:04.201472    2577 projected.go:196] Error preparing data for projected volume kube-api-access-vl5jc for pod openshift-network-diagnostics/network-check-target-d9xr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:04.201523 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:04.201512    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret podName:f399aaa3-b634-40e9-9ca7-0941f0ec2925 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:06.201498147 +0000 UTC m=+6.210846413 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret") pod "global-pull-secret-syncer-jkz4p" (UID: "f399aaa3-b634-40e9-9ca7-0941f0ec2925") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:04.201523 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:04.201529    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc podName:b9a64b7e-6436-4415-86f9-735350f24e35 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:06.201521562 +0000 UTC m=+6.210869828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vl5jc" (UniqueName: "kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc") pod "network-check-target-d9xr4" (UID: "b9a64b7e-6436-4415-86f9-735350f24e35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:04.451658 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:04.451613    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:04.451821 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:04.451686    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:04.451821 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:04.451702    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:04.451939 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:04.451817    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xr4" podUID="b9a64b7e-6436-4415-86f9-735350f24e35"
Oct 30 20:19:04.451939 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:04.451923    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkz4p" podUID="f399aaa3-b634-40e9-9ca7-0941f0ec2925"
Oct 30 20:19:04.452044 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:04.452008    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkj2j" podUID="c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256"
Oct 30 20:19:04.624183 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:04.624113    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:14:03 +0000 UTC" deadline="2028-06-28 08:20:48.347949107 +0000 UTC"
Oct 30 20:19:04.624183 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:04.624182    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14556h1m43.723771866s"
Oct 30 20:19:06.222140 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:06.221827    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl5jc\" (UniqueName: \"kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc\") pod \"network-check-target-d9xr4\" (UID: \"b9a64b7e-6436-4415-86f9-735350f24e35\") " pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:06.222657 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:06.222141    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:06.222657 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:06.221969    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:19:06.222657 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:06.222233    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:19:06.222657 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:06.222248    2577 projected.go:196] Error preparing data for projected volume kube-api-access-vl5jc for pod openshift-network-diagnostics/network-check-target-d9xr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:06.222657 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:06.222264    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:06.222657 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:06.222312    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc podName:b9a64b7e-6436-4415-86f9-735350f24e35 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:10.222288997 +0000 UTC m=+10.231637298 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vl5jc" (UniqueName: "kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc") pod "network-check-target-d9xr4" (UID: "b9a64b7e-6436-4415-86f9-735350f24e35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:06.222657 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:06.222355    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:06.222657 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:06.222342    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs podName:c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:10.222334126 +0000 UTC m=+10.231682391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs") pod "network-metrics-daemon-vkj2j" (UID: "c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:06.222657 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:06.222178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs\") pod \"network-metrics-daemon-vkj2j\" (UID: \"c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256\") " pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:06.222657 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:06.222393    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret podName:f399aaa3-b634-40e9-9ca7-0941f0ec2925 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:10.222377681 +0000 UTC m=+10.231725961 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret") pod "global-pull-secret-syncer-jkz4p" (UID: "f399aaa3-b634-40e9-9ca7-0941f0ec2925") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:06.447546 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:06.447514    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:06.447700 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:06.447556    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:06.447700 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:06.447663    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkz4p" podUID="f399aaa3-b634-40e9-9ca7-0941f0ec2925"
Oct 30 20:19:06.447700 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:06.447520    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:06.447870 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:06.447792    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkj2j" podUID="c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256"
Oct 30 20:19:06.447870 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:06.447857    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xr4" podUID="b9a64b7e-6436-4415-86f9-735350f24e35"
Oct 30 20:19:06.485992 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:06.485921    2577 generic.go:356] "Generic (PLEG): container finished" podID="be68dc42ad591cdc0bc8a690a67f92fa" containerID="6b9eddcb6f4d3460656b0c47cbe1eb4c9b4a9c9c11928de2c64a93041e6cc070" exitCode=0
Oct 30 20:19:06.486128 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:06.486026    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal" event={"ID":"be68dc42ad591cdc0bc8a690a67f92fa","Type":"ContainerDied","Data":"6b9eddcb6f4d3460656b0c47cbe1eb4c9b4a9c9c11928de2c64a93041e6cc070"}
Oct 30 20:19:06.497246 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:06.496707    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal" event={"ID":"d10e19b03ae8d13f3ef8ccaa1c90976f","Type":"ContainerStarted","Data":"0fa413ed30968268f16a6156e0da49568bdcba923c52aa59fdd97f8beb36da89"}
Oct 30 20:19:07.510759 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:07.510675    2577 generic.go:356] "Generic (PLEG): container finished" podID="51727502-b532-46d0-828c-140b8eda2ba3" containerID="fddd934d827a804b7782856ffbb44761ae631b71df69bbf5e3ccb22897b1e0c4" exitCode=0
Oct 30 20:19:07.511287 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:07.510788    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tgdb" event={"ID":"51727502-b532-46d0-828c-140b8eda2ba3","Type":"ContainerDied","Data":"fddd934d827a804b7782856ffbb44761ae631b71df69bbf5e3ccb22897b1e0c4"}
Oct 30 20:19:07.518267 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:07.518089    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal" event={"ID":"be68dc42ad591cdc0bc8a690a67f92fa","Type":"ContainerStarted","Data":"3b9109434d6aacb41b5881ddf431803ffcf6fbada1ba286a6d45856a8de1f133"}
Oct 30 20:19:07.758264 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:07.758200    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal" podStartSLOduration=5.758179409 podStartE2EDuration="5.758179409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:06.741207067 +0000 UTC m=+6.750555358" watchObservedRunningTime="2026-10-30 20:19:07.758179409 +0000 UTC m=+7.767527698"
Oct 30 20:19:08.449547 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:08.449508    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:08.449700 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:08.449670    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkj2j" podUID="c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256"
Oct 30 20:19:08.449781 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:08.449765    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:08.449880 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:08.449857    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkz4p" podUID="f399aaa3-b634-40e9-9ca7-0941f0ec2925"
Oct 30 20:19:08.449943 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:08.449929    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:08.450030 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:08.450011    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xr4" podUID="b9a64b7e-6436-4415-86f9-735350f24e35"
Oct 30 20:19:08.524580 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:08.524484    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qxwhp" event={"ID":"1089345a-f93e-4c78-ad1c-ac38961bdf10","Type":"ContainerStarted","Data":"163b8315f06d86df20d4125551620f7b643eea88938058aa37a358944f1c5089"}
Oct 30 20:19:08.654180 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:08.654119    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal" podStartSLOduration=6.6540975620000005 podStartE2EDuration="6.654097562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:07.849259775 +0000 UTC m=+7.858608067" watchObservedRunningTime="2026-10-30 20:19:08.654097562 +0000 UTC m=+8.663445852"
Oct 30 20:19:09.531295 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:09.531243    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n8gtv" event={"ID":"f82dd16e-0e8a-40ef-87bd-84560a6b9b71","Type":"ContainerStarted","Data":"4731441c973ffe4a5433ff03362899ab4f2db73b5e7de76516852bf8a5a3a70c"}
Oct 30 20:19:09.662597 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:09.662494    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qxwhp" podStartSLOduration=3.921702344 podStartE2EDuration="8.6624717s" totalImagesPullingTime="4.740769356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:08.65515348 +0000 UTC m=+8.664501770" watchObservedRunningTime="2026-10-30 20:19:09.6624717 +0000 UTC m=+9.671819990"
Oct 30 20:19:09.663203 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:09.663172    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n8gtv" podStartSLOduration=2.86322281 podStartE2EDuration="8.663158634s" totalImagesPullingTime="5.799935824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:09.658295359 +0000 UTC m=+9.667643647" watchObservedRunningTime="2026-10-30 20:19:09.663158634 +0000 UTC m=+9.672506942"
Oct 30 20:19:10.260489 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:10.260427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:10.260489 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:10.260487    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs\") pod \"network-metrics-daemon-vkj2j\" (UID: \"c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256\") " pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:10.260735 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:10.260585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl5jc\" (UniqueName: \"kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc\") pod \"network-check-target-d9xr4\" (UID: \"b9a64b7e-6436-4415-86f9-735350f24e35\") " pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:10.260796 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:10.260779    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:19:10.260847 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:10.260799    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:19:10.260847 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:10.260813    2577 projected.go:196] Error preparing data for projected volume kube-api-access-vl5jc for pod openshift-network-diagnostics/network-check-target-d9xr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:10.260907 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:10.260875    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc podName:b9a64b7e-6436-4415-86f9-735350f24e35 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:18.260854495 +0000 UTC m=+18.270202780 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vl5jc" (UniqueName: "kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc") pod "network-check-target-d9xr4" (UID: "b9a64b7e-6436-4415-86f9-735350f24e35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:10.261308 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:10.260977    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:10.261308 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:10.261038    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret podName:f399aaa3-b634-40e9-9ca7-0941f0ec2925 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:18.261025459 +0000 UTC m=+18.270373739 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret") pod "global-pull-secret-syncer-jkz4p" (UID: "f399aaa3-b634-40e9-9ca7-0941f0ec2925") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:10.261308 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:10.261122    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:10.261308 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:10.261205    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs podName:c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:18.261192586 +0000 UTC m=+18.270540866 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs") pod "network-metrics-daemon-vkj2j" (UID: "c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:10.452453 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:10.451624    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:10.452453 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:10.451746    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xr4" podUID="b9a64b7e-6436-4415-86f9-735350f24e35"
Oct 30 20:19:10.452453 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:10.451826    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:10.452453 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:10.451915    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkj2j" podUID="c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256"
Oct 30 20:19:10.452453 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:10.451967    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:10.452453 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:10.452039    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkz4p" podUID="f399aaa3-b634-40e9-9ca7-0941f0ec2925"
Oct 30 20:19:10.535328 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:10.535226    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg" event={"ID":"3e789fe9-f262-4147-b4ee-ec42835f4ed0","Type":"ContainerStarted","Data":"44baf31ac037e99703a6e8809fb4a56d7b0ae352e4a4b24a9c6bd201da7fd59d"}
Oct 30 20:19:11.540666 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:11.540022    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-psqxp" event={"ID":"0a0b6cf3-8303-4d98-b22f-f8a313721702","Type":"ContainerStarted","Data":"622326ab51da379afdc9a69e709b6c93e7c18bbc1ab7cf59ca11ee68959f258b"}
Oct 30 20:19:11.663179 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:11.663151    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 20:19:12.448952 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:12.448502    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:12.448952 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:12.448629    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkj2j" podUID="c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256"
Oct 30 20:19:12.448952 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:12.448696    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:12.448952 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:12.448756    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkz4p" podUID="f399aaa3-b634-40e9-9ca7-0941f0ec2925"
Oct 30 20:19:12.448952 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:12.448826    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:12.448952 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:12.448916    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xr4" podUID="b9a64b7e-6436-4415-86f9-735350f24e35"
Oct 30 20:19:12.454208 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:12.454107    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:19:11.663176114Z","UUID":"f9436227-f7f0-4089-ab2c-2353939896c4","Handler":null,"Name":"","Endpoint":""}
Oct 30 20:19:12.456082 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:12.456060    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 20:19:12.456200 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:12.456089    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 20:19:12.547883 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:12.547804    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tgdb" event={"ID":"51727502-b532-46d0-828c-140b8eda2ba3","Type":"ContainerStarted","Data":"cccb4b8a44f0dd3a1665d9ca372611bf562f31f0ea10d178af06e110bfc66f25"}
Oct 30 20:19:12.550118 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:12.550085    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jbb5l" event={"ID":"bd71c9e1-948b-41cc-9d05-a65a556c8ea7","Type":"ContainerStarted","Data":"8e175a3f62a72e23c36f6cccc63b4333eed7b163fc6ef0751b2bc7916d17e34f"}
Oct 30 20:19:12.553234 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:12.553207    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg" event={"ID":"3e789fe9-f262-4147-b4ee-ec42835f4ed0","Type":"ContainerStarted","Data":"312f245d279548242f4bdd744ac6bfe41ad1485cb71862aaebc88800beba7da3"}
Oct 30 20:19:12.692316 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:12.692259    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-psqxp" podStartSLOduration=3.095514488 podStartE2EDuration="11.692244424s" totalImagesPullingTime="8.596729936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:11.6764587 +0000 UTC m=+11.685806993" watchObservedRunningTime="2026-10-30 20:19:12.692244424 +0000 UTC m=+12.701592713"
Oct 30 20:19:12.992200 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:12.991930    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qxwhp"
Oct 30 20:19:12.993059 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:12.993032    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qxwhp"
Oct 30 20:19:13.122406 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:13.122325    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-jbb5l" podStartSLOduration=3.084979255 podStartE2EDuration="12.122303361s" totalImagesPullingTime="9.037324106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:12.820709145 +0000 UTC m=+12.830057432" watchObservedRunningTime="2026-10-30 20:19:13.122303361 +0000 UTC m=+13.131651650"
Oct 30 20:19:13.558029 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:13.557875    2577 generic.go:356] "Generic (PLEG): container finished" podID="51727502-b532-46d0-828c-140b8eda2ba3" containerID="cccb4b8a44f0dd3a1665d9ca372611bf562f31f0ea10d178af06e110bfc66f25" exitCode=0
Oct 30 20:19:13.558656 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:13.558161    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tgdb" event={"ID":"51727502-b532-46d0-828c-140b8eda2ba3","Type":"ContainerDied","Data":"cccb4b8a44f0dd3a1665d9ca372611bf562f31f0ea10d178af06e110bfc66f25"}
Oct 30 20:19:13.560232 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:13.560020    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r7mrf" event={"ID":"c85a95c6-4b03-47db-a7a3-f702a80fe01f","Type":"ContainerStarted","Data":"913f566ddf09fb2436849f5d5926db8a095475d25afdd9f59a12f9dfd75a9f3c"}
Oct 30 20:19:13.564847 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:13.564770    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg" event={"ID":"3e789fe9-f262-4147-b4ee-ec42835f4ed0","Type":"ContainerStarted","Data":"36e110ccdd960f4283bc797c9ae59c123c8843545572464bc62eac5177ea102a"}
Oct 30 20:19:13.844186 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:13.844085    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg" podStartSLOduration=2.879917265 podStartE2EDuration="12.844066567s" totalImagesPullingTime="9.964149302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:19:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:13.841700475 +0000 UTC m=+13.851048750" watchObservedRunningTime="2026-10-30 20:19:13.844066567 +0000 UTC m=+13.853414857"
Oct 30 20:19:13.966202 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:13.966136    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r7mrf" podStartSLOduration=4.282105961 podStartE2EDuration="12.966113417s" totalImagesPullingTime="8.684007456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:13.965873831 +0000 UTC m=+13.975222124" watchObservedRunningTime="2026-10-30 20:19:13.966113417 +0000 UTC m=+13.975461709"
Oct 30 20:19:14.447262 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:14.447226    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:14.447435 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:14.447301    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:14.447435 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:14.447298    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:14.447562 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:14.447438    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xr4" podUID="b9a64b7e-6436-4415-86f9-735350f24e35"
Oct 30 20:19:14.447562 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:14.447508    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkz4p" podUID="f399aaa3-b634-40e9-9ca7-0941f0ec2925"
Oct 30 20:19:14.447562 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:14.447536    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkj2j" podUID="c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256"
Oct 30 20:19:14.569402 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:14.569358    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cwm29" event={"ID":"5f9aa5d5-0be0-4914-a6cc-bd54151feb49","Type":"ContainerStarted","Data":"05565c36c58acbc5512f6a643b7fab3bdc77ef4dca231890121ebae00344e08b"}
Oct 30 20:19:15.573858 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:15.573811    2577 generic.go:356] "Generic (PLEG): container finished" podID="51727502-b532-46d0-828c-140b8eda2ba3" containerID="8741dd37a60df0706d2c734dee0bc3d2957a50825e45a6e73f9154086ab991c8" exitCode=0
Oct 30 20:19:15.574510 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:15.573876    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tgdb" event={"ID":"51727502-b532-46d0-828c-140b8eda2ba3","Type":"ContainerDied","Data":"8741dd37a60df0706d2c734dee0bc3d2957a50825e45a6e73f9154086ab991c8"}
Oct 30 20:19:15.699515 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:15.699469    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cwm29" podStartSLOduration=3.689626094 podStartE2EDuration="14.699455417s" totalImagesPullingTime="11.009829323s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:14.707138645 +0000 UTC m=+14.716486934" watchObservedRunningTime="2026-10-30 20:19:15.699455417 +0000 UTC m=+15.708803701"
Oct 30 20:19:16.447019 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:16.446988    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:16.447186 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:16.447098    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkz4p" podUID="f399aaa3-b634-40e9-9ca7-0941f0ec2925"
Oct 30 20:19:16.447186 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:16.446994    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:16.447186 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:16.447181    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkj2j" podUID="c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256"
Oct 30 20:19:16.447336 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:16.446988    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:16.447336 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:16.447230    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xr4" podUID="b9a64b7e-6436-4415-86f9-735350f24e35"
Oct 30 20:19:17.582239 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:17.582022    2577 generic.go:356] "Generic (PLEG): container finished" podID="51727502-b532-46d0-828c-140b8eda2ba3" containerID="f60c093d4f453449d33b82097be4951c215229dacb03cafde45ec40297f2f6d3" exitCode=0
Oct 30 20:19:17.582809 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:17.582101    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tgdb" event={"ID":"51727502-b532-46d0-828c-140b8eda2ba3","Type":"ContainerDied","Data":"f60c093d4f453449d33b82097be4951c215229dacb03cafde45ec40297f2f6d3"}
Oct 30 20:19:18.325974 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:18.325945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:18.325974 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:18.325980    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs\") pod \"network-metrics-daemon-vkj2j\" (UID: \"c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256\") " pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:18.326147 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:18.326020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl5jc\" (UniqueName: \"kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc\") pod \"network-check-target-d9xr4\" (UID: \"b9a64b7e-6436-4415-86f9-735350f24e35\") " pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:18.326147 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:18.326086    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:18.326147 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:18.326110    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:19:18.326147 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:18.326126    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:19:18.326147 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:18.326136    2577 projected.go:196] Error preparing data for projected volume kube-api-access-vl5jc for pod openshift-network-diagnostics/network-check-target-d9xr4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:18.326147 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:18.326140    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:18.326314 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:18.326149    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret podName:f399aaa3-b634-40e9-9ca7-0941f0ec2925 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:34.326131158 +0000 UTC m=+34.335479439 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret") pod "global-pull-secret-syncer-jkz4p" (UID: "f399aaa3-b634-40e9-9ca7-0941f0ec2925") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:19:18.326314 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:18.326196    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc podName:b9a64b7e-6436-4415-86f9-735350f24e35 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:34.32618488 +0000 UTC m=+34.335533145 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vl5jc" (UniqueName: "kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc") pod "network-check-target-d9xr4" (UID: "b9a64b7e-6436-4415-86f9-735350f24e35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:19:18.326314 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:18.326207    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs podName:c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:34.326201377 +0000 UTC m=+34.335549640 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs") pod "network-metrics-daemon-vkj2j" (UID: "c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:19:18.447410 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:18.447338    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:18.447410 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:18.447405    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:18.447410 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:18.447407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:18.447598 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:18.447491    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xr4" podUID="b9a64b7e-6436-4415-86f9-735350f24e35"
Oct 30 20:19:18.447598 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:18.447554    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkj2j" podUID="c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256"
Oct 30 20:19:18.447669 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:18.447638    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkz4p" podUID="f399aaa3-b634-40e9-9ca7-0941f0ec2925"
Oct 30 20:19:18.588360 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:18.588308    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frx59" event={"ID":"fb857965-3140-4b4e-982c-af2680bbeff3","Type":"ContainerStarted","Data":"c2b8904b81965ef387800611af6e09819d01f5b5abaa947becb69e5e3533f27f"}
Oct 30 20:19:18.588800 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:18.588408    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frx59" event={"ID":"fb857965-3140-4b4e-982c-af2680bbeff3","Type":"ContainerStarted","Data":"5813d5705dc67084dbd48254ab5ce71d73026f04ba70d55214730a4cce5171ef"}
Oct 30 20:19:18.588800 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:18.588457    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frx59" event={"ID":"fb857965-3140-4b4e-982c-af2680bbeff3","Type":"ContainerStarted","Data":"6a9b1098853db9e0d184745443ae4e3d4a87d5918feb43495830e555bce54db0"}
Oct 30 20:19:18.588800 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:18.588504    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frx59" event={"ID":"fb857965-3140-4b4e-982c-af2680bbeff3","Type":"ContainerStarted","Data":"5b0fff87d9c5a17f18df0fd07c86d4c1da6374b088a51a6e556a0ffa6e201062"}
Oct 30 20:19:18.588800 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:18.588547    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frx59" event={"ID":"fb857965-3140-4b4e-982c-af2680bbeff3","Type":"ContainerStarted","Data":"155d95f96aa3e94735240c0b48103dd1c37e1b8fbe96e172ce095cb7fd0b9559"}
Oct 30 20:19:18.588800 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:18.588589    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frx59" event={"ID":"fb857965-3140-4b4e-982c-af2680bbeff3","Type":"ContainerStarted","Data":"150f738ba5057aef38239723ac69d0b48273109b9ee38b5e980e3153e7b4efaf"}
Oct 30 20:19:19.891554 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:19.891031    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qxwhp"
Oct 30 20:19:19.891554 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:19.891379    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:19:19.891917 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:19.891583    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qxwhp"
Oct 30 20:19:20.448424 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:20.448401    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:20.448570 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:20.448494    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:20.448570 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:20.448507    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkz4p" podUID="f399aaa3-b634-40e9-9ca7-0941f0ec2925"
Oct 30 20:19:20.448570 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:20.448517    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:20.448688 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:20.448588    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkj2j" podUID="c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256"
Oct 30 20:19:20.448723 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:20.448692    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xr4" podUID="b9a64b7e-6436-4415-86f9-735350f24e35"
Oct 30 20:19:20.595187 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:20.595108    2577 generic.go:356] "Generic (PLEG): container finished" podID="51727502-b532-46d0-828c-140b8eda2ba3" containerID="eb4e0f901d3c9e537e39e21b6df47da7aa2b4ec3f095981bde0af8f118fbb494" exitCode=0
Oct 30 20:19:20.595363 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:20.595191    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tgdb" event={"ID":"51727502-b532-46d0-828c-140b8eda2ba3","Type":"ContainerDied","Data":"eb4e0f901d3c9e537e39e21b6df47da7aa2b4ec3f095981bde0af8f118fbb494"}
Oct 30 20:19:20.598462 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:20.598441    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frx59" event={"ID":"fb857965-3140-4b4e-982c-af2680bbeff3","Type":"ContainerStarted","Data":"5310bb876b6a17d0525a592008b6acbd6bc4d1a497b65176ffc080220295ead0"}
Oct 30 20:19:21.603403 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:21.603374    2577 generic.go:356] "Generic (PLEG): container finished" podID="51727502-b532-46d0-828c-140b8eda2ba3" containerID="29e122e5925d2c60fa39954ab631634008f021c9900278c275bcb2dc1cfe7846" exitCode=0
Oct 30 20:19:21.603912 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:21.603432    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tgdb" event={"ID":"51727502-b532-46d0-828c-140b8eda2ba3","Type":"ContainerDied","Data":"29e122e5925d2c60fa39954ab631634008f021c9900278c275bcb2dc1cfe7846"}
Oct 30 20:19:22.447186 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:22.447156    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:22.447291 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:22.447189    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:22.447291 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:22.447258    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkj2j" podUID="c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256"
Oct 30 20:19:22.447400 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:22.447310    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:22.447471 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:22.447446    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkz4p" podUID="f399aaa3-b634-40e9-9ca7-0941f0ec2925"
Oct 30 20:19:22.447560 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:22.447542    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xr4" podUID="b9a64b7e-6436-4415-86f9-735350f24e35"
Oct 30 20:19:22.608618 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:22.608585    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frx59" event={"ID":"fb857965-3140-4b4e-982c-af2680bbeff3","Type":"ContainerStarted","Data":"acc136a360b5012585a2f983e7654b3a1a4c30e8cfd31dedaf46c6e1c0fefe49"}
Oct 30 20:19:22.609151 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:22.608834    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:22.609151 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:22.608889    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:22.609151 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:22.608925    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:22.612578 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:22.612551    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tgdb" event={"ID":"51727502-b532-46d0-828c-140b8eda2ba3","Type":"ContainerStarted","Data":"3d88fd6a2e5a84ffc855bae316f82b5ffa6c82abdfae487a013068ac4e1a68b6"}
Oct 30 20:19:22.623885 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:22.623867    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:22.624101 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:22.623961    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:22.874276 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:22.874182    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-frx59" podStartSLOduration=6.8149679899999995 podStartE2EDuration="21.874167417s" totalImagesPullingTime="15.059199427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:22.835939862 +0000 UTC m=+22.845288146" watchObservedRunningTime="2026-10-30 20:19:22.874167417 +0000 UTC m=+22.883515701"
Oct 30 20:19:23.095430 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:23.095376    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2tgdb" podStartSLOduration=9.250940996 podStartE2EDuration="22.0953633s" totalImagesPullingTime="12.844422304s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:19:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:23.094760872 +0000 UTC m=+23.104109180" watchObservedRunningTime="2026-10-30 20:19:23.0953633 +0000 UTC m=+23.104711584"
Oct 30 20:19:24.447606 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:24.447570    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:24.448064 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:24.447585    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:24.448064 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:24.447690    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:24.448064 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:24.447691    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkz4p" podUID="f399aaa3-b634-40e9-9ca7-0941f0ec2925"
Oct 30 20:19:24.448064 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:24.447771    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkj2j" podUID="c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256"
Oct 30 20:19:24.448064 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:24.447861    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xr4" podUID="b9a64b7e-6436-4415-86f9-735350f24e35"
Oct 30 20:19:25.360284 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:25.360078    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vkj2j"]
Oct 30 20:19:25.360463 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:25.360334    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:25.360463 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:25.360427    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkj2j" podUID="c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256"
Oct 30 20:19:25.410557 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:25.410531    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jkz4p"]
Oct 30 20:19:25.410665 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:25.410619    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:25.410734 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:25.410714    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkz4p" podUID="f399aaa3-b634-40e9-9ca7-0941f0ec2925"
Oct 30 20:19:25.411082 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:25.411067    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d9xr4"]
Oct 30 20:19:25.411135 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:25.411125    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:25.411197 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:25.411183    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xr4" podUID="b9a64b7e-6436-4415-86f9-735350f24e35"
Oct 30 20:19:27.447813 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:27.447783    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:27.448139 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:27.447859    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:27.448139 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:27.447940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:27.448139 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:27.447947    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xr4" podUID="b9a64b7e-6436-4415-86f9-735350f24e35"
Oct 30 20:19:27.448139 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:27.447999    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkz4p" podUID="f399aaa3-b634-40e9-9ca7-0941f0ec2925"
Oct 30 20:19:27.448139 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:27.448085    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkj2j" podUID="c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256"
Oct 30 20:19:29.447342 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:29.447308    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:29.447718 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:29.447309    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:29.447718 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:29.447411    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d9xr4" podUID="b9a64b7e-6436-4415-86f9-735350f24e35"
Oct 30 20:19:29.447718 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:29.447309    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:29.447718 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:29.447499    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vkj2j" podUID="c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256"
Oct 30 20:19:29.447718 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:29.447584    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkz4p" podUID="f399aaa3-b634-40e9-9ca7-0941f0ec2925"
Oct 30 20:19:30.181521 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.181493    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeReady"
Oct 30 20:19:30.181667 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.181627    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 20:19:30.304304 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.304279    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mzdm7"]
Oct 30 20:19:30.305816 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.305803    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6pstp"]
Oct 30 20:19:30.305927 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.305913    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mzdm7"
Oct 30 20:19:30.307626 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.307607    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:30.321222 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.321201    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-m8znf-jgfmg"]
Oct 30 20:19:30.322613 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.322597    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:19:30.340810 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.340766    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jlxjn"]
Oct 30 20:19:30.342216 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.342204    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:30.358796 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.358770    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mzdm7"]
Oct 30 20:19:30.362261 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.362222    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 20:19:30.362261 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.362220    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 20:19:30.362411 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.362277    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 20:19:30.376671 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.376649    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6pstp"]
Oct 30 20:19:30.384600 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.384585    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 20:19:30.384865 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.384851    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 20:19:30.390764 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.390748    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bp5wk\""
Oct 30 20:19:30.408430 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.408413    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f4ct2\""
Oct 30 20:19:30.408430 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.408419    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q6sjr\""
Oct 30 20:19:30.412694 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.412679    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 20:19:30.412750 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.412718    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vdqtm\""
Oct 30 20:19:30.414964 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.414948    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nfhx\" (UniqueName: \"kubernetes.io/projected/cb8a13f8-5b1b-4a9d-965f-ad3e29051484-kube-api-access-6nfhx\") pod \"ingress-canary-mzdm7\" (UID: \"cb8a13f8-5b1b-4a9d-965f-ad3e29051484\") " pod="openshift-ingress-canary/ingress-canary-mzdm7"
Oct 30 20:19:30.415009 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.414973    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09961adc-4403-44ec-8c11-81938ae4fb16-service-ca-bundle\") pod \"periodic-gathering-m8znf-jgfmg\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") " pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:19:30.415009 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.414990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5ab344-e8d9-4b66-8e1f-6283db623632-tmp-dir\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:30.415080 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.415017    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m5xt\" (UniqueName: \"kubernetes.io/projected/09961adc-4403-44ec-8c11-81938ae4fb16-kube-api-access-4m5xt\") pod \"periodic-gathering-m8znf-jgfmg\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") " pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:19:30.415080 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.415052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:30.415154 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.415095    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:30.415154 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.415131    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:30.415154 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.415149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wgv9\" (UniqueName: \"kubernetes.io/projected/4d5ab344-e8d9-4b66-8e1f-6283db623632-kube-api-access-7wgv9\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:30.415254 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.415181    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5ab344-e8d9-4b66-8e1f-6283db623632-config-volume\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:30.415254 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.415196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb8a13f8-5b1b-4a9d-965f-ad3e29051484-cert\") pod \"ingress-canary-mzdm7\" (UID: \"cb8a13f8-5b1b-4a9d-965f-ad3e29051484\") " pod="openshift-ingress-canary/ingress-canary-mzdm7"
Oct 30 20:19:30.415254 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.415219    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98a309d0-a2a2-4ec4-82c4-5966823b6e06-data-volume\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:30.415254 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.415239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09961adc-4403-44ec-8c11-81938ae4fb16-trusted-ca-bundle\") pod \"periodic-gathering-m8znf-jgfmg\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") " pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:19:30.415420 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.415302    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98a309d0-a2a2-4ec4-82c4-5966823b6e06-crio-socket\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:30.415420 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.415346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms86z\" (UniqueName: \"kubernetes.io/projected/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-api-access-ms86z\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:30.415420 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.415367    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/09961adc-4403-44ec-8c11-81938ae4fb16-archives-path\") pod \"periodic-gathering-m8znf-jgfmg\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") " pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:19:30.444190 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.444165    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-m8znf-jgfmg"]
Oct 30 20:19:30.444867 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.444847    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jlxjn"]
Oct 30 20:19:30.460385 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.460364    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 20:19:30.460671 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.460394    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 20:19:30.460671 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.460397    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 20:19:30.460671 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.460415    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 20:19:30.511629 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.511613    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 20:19:30.515790 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.515775    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m5xt\" (UniqueName: \"kubernetes.io/projected/09961adc-4403-44ec-8c11-81938ae4fb16-kube-api-access-4m5xt\") pod \"periodic-gathering-m8znf-jgfmg\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") " pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:19:30.515837 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.515800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:30.515837 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.515816    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:30.515837 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.515832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:30.515970 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.515847    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wgv9\" (UniqueName: \"kubernetes.io/projected/4d5ab344-e8d9-4b66-8e1f-6283db623632-kube-api-access-7wgv9\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:30.515970 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:30.515897    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 20:19:30.515970 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.515925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5ab344-e8d9-4b66-8e1f-6283db623632-config-volume\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:30.515970 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:30.515931    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:19:30.515970 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:30.515965    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-rbac-proxy-cm podName:98a309d0-a2a2-4ec4-82c4-5966823b6e06 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:31.015950622 +0000 UTC m=+31.025298905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6pstp" (UID: "98a309d0-a2a2-4ec4-82c4-5966823b6e06") : configmap "kube-rbac-proxy" not found
Oct 30 20:19:30.516204 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:30.515984    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls podName:4d5ab344-e8d9-4b66-8e1f-6283db623632 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:31.015973068 +0000 UTC m=+31.025321331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls") pod "dns-default-jlxjn" (UID: "4d5ab344-e8d9-4b66-8e1f-6283db623632") : secret "dns-default-metrics-tls" not found
Oct 30 20:19:30.516204 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.515995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb8a13f8-5b1b-4a9d-965f-ad3e29051484-cert\") pod \"ingress-canary-mzdm7\" (UID: \"cb8a13f8-5b1b-4a9d-965f-ad3e29051484\") " pod="openshift-ingress-canary/ingress-canary-mzdm7"
Oct 30 20:19:30.516204 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:30.515936    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:30.516204 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.516020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98a309d0-a2a2-4ec4-82c4-5966823b6e06-data-volume\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:30.516204 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.516037    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09961adc-4403-44ec-8c11-81938ae4fb16-trusted-ca-bundle\") pod \"periodic-gathering-m8znf-jgfmg\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") " pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:19:30.516204 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:30.516061    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls podName:98a309d0-a2a2-4ec4-82c4-5966823b6e06 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:31.016048339 +0000 UTC m=+31.025396625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6pstp" (UID: "98a309d0-a2a2-4ec4-82c4-5966823b6e06") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:30.516204 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.516095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98a309d0-a2a2-4ec4-82c4-5966823b6e06-crio-socket\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:30.516204 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.516141    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms86z\" (UniqueName: \"kubernetes.io/projected/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-api-access-ms86z\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:30.516204 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.516168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/09961adc-4403-44ec-8c11-81938ae4fb16-archives-path\") pod \"periodic-gathering-m8znf-jgfmg\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") " pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:19:30.516597 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.516216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nfhx\" (UniqueName: \"kubernetes.io/projected/cb8a13f8-5b1b-4a9d-965f-ad3e29051484-kube-api-access-6nfhx\") pod \"ingress-canary-mzdm7\" (UID: \"cb8a13f8-5b1b-4a9d-965f-ad3e29051484\") " pod="openshift-ingress-canary/ingress-canary-mzdm7"
Oct 30 20:19:30.516597 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.516240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09961adc-4403-44ec-8c11-81938ae4fb16-service-ca-bundle\") pod \"periodic-gathering-m8znf-jgfmg\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") " pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:19:30.516597 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.516265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5ab344-e8d9-4b66-8e1f-6283db623632-tmp-dir\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:30.516597 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.516283    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98a309d0-a2a2-4ec4-82c4-5966823b6e06-crio-socket\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:30.516597 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.516563    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5ab344-e8d9-4b66-8e1f-6283db623632-config-volume\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:30.516735 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.516611    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5ab344-e8d9-4b66-8e1f-6283db623632-tmp-dir\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:30.516767 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.516730    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/09961adc-4403-44ec-8c11-81938ae4fb16-archives-path\") pod \"periodic-gathering-m8znf-jgfmg\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") " pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:19:30.516891 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.516875    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98a309d0-a2a2-4ec4-82c4-5966823b6e06-data-volume\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:30.516922 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.516884    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09961adc-4403-44ec-8c11-81938ae4fb16-service-ca-bundle\") pod \"periodic-gathering-m8znf-jgfmg\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") " pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:19:30.516951 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.516925    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09961adc-4403-44ec-8c11-81938ae4fb16-trusted-ca-bundle\") pod \"periodic-gathering-m8znf-jgfmg\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") " pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:19:30.520151 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.520048    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb8a13f8-5b1b-4a9d-965f-ad3e29051484-cert\") pod \"ingress-canary-mzdm7\" (UID: \"cb8a13f8-5b1b-4a9d-965f-ad3e29051484\") " pod="openshift-ingress-canary/ingress-canary-mzdm7"
Oct 30 20:19:30.575666 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.575641    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m5xt\" (UniqueName: \"kubernetes.io/projected/09961adc-4403-44ec-8c11-81938ae4fb16-kube-api-access-4m5xt\") pod \"periodic-gathering-m8znf-jgfmg\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") " pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:19:30.575838 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.575818    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wgv9\" (UniqueName: \"kubernetes.io/projected/4d5ab344-e8d9-4b66-8e1f-6283db623632-kube-api-access-7wgv9\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:30.576608 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.576579    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms86z\" (UniqueName: \"kubernetes.io/projected/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-api-access-ms86z\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:30.576757 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.576743    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nfhx\" (UniqueName: \"kubernetes.io/projected/cb8a13f8-5b1b-4a9d-965f-ad3e29051484-kube-api-access-6nfhx\") pod \"ingress-canary-mzdm7\" (UID: \"cb8a13f8-5b1b-4a9d-965f-ad3e29051484\") " pod="openshift-ingress-canary/ingress-canary-mzdm7"
Oct 30 20:19:30.616990 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.616933    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mzdm7"
Oct 30 20:19:30.631405 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:30.631388    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:19:31.017528 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.017458    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-m8znf-jgfmg"]
Oct 30 20:19:31.020047 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.020025    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:31.020125 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.020062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:31.020125 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.020089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:31.020217 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:31.020154    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 20:19:31.020217 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:31.020190    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:19:31.020301 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:31.020209    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:31.020301 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:31.020223    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-rbac-proxy-cm podName:98a309d0-a2a2-4ec4-82c4-5966823b6e06 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:32.020208878 +0000 UTC m=+32.029557157 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6pstp" (UID: "98a309d0-a2a2-4ec4-82c4-5966823b6e06") : configmap "kube-rbac-proxy" not found
Oct 30 20:19:31.020301 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:31.020244    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls podName:4d5ab344-e8d9-4b66-8e1f-6283db623632 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:32.020237068 +0000 UTC m=+32.029585359 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls") pod "dns-default-jlxjn" (UID: "4d5ab344-e8d9-4b66-8e1f-6283db623632") : secret "dns-default-metrics-tls" not found
Oct 30 20:19:31.020301 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:31.020261    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls podName:98a309d0-a2a2-4ec4-82c4-5966823b6e06 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:32.020253016 +0000 UTC m=+32.029601283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6pstp" (UID: "98a309d0-a2a2-4ec4-82c4-5966823b6e06") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:31.020536 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:31.020520    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09961adc_4403_44ec_8c11_81938ae4fb16.slice/crio-ad5afd8493f2d8587877e3d0b6dacc834677ef236c98caae8deb086eb44ac13a WatchSource:0}: Error finding container ad5afd8493f2d8587877e3d0b6dacc834677ef236c98caae8deb086eb44ac13a: Status 404 returned error can't find the container with id ad5afd8493f2d8587877e3d0b6dacc834677ef236c98caae8deb086eb44ac13a
Oct 30 20:19:31.065902 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.065884    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mzdm7"]
Oct 30 20:19:31.067719 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:31.067694    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8a13f8_5b1b_4a9d_965f_ad3e29051484.slice/crio-8503ccb525b56b73d936998fb4aebdf71737084bc8df057d8585b7700a320f01 WatchSource:0}: Error finding container 8503ccb525b56b73d936998fb4aebdf71737084bc8df057d8585b7700a320f01: Status 404 returned error can't find the container with id 8503ccb525b56b73d936998fb4aebdf71737084bc8df057d8585b7700a320f01
Oct 30 20:19:31.447558 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.447481    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:31.447558 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.447514    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:31.447774 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.447735    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:31.506350 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.506312    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 20:19:31.506857 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.506459    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 20:19:31.507887 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.507870    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 20:19:31.507976 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.507900    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cxq5f\""
Oct 30 20:19:31.507976 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.507886    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wz6nt\""
Oct 30 20:19:31.507976 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.507924    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 20:19:31.507976 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.507923    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nnssb\""
Oct 30 20:19:31.630205 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.630176    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-m8znf-jgfmg" event={"ID":"09961adc-4403-44ec-8c11-81938ae4fb16","Type":"ContainerStarted","Data":"ad5afd8493f2d8587877e3d0b6dacc834677ef236c98caae8deb086eb44ac13a"}
Oct 30 20:19:31.631282 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.631263    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mzdm7" event={"ID":"cb8a13f8-5b1b-4a9d-965f-ad3e29051484","Type":"ContainerStarted","Data":"8503ccb525b56b73d936998fb4aebdf71737084bc8df057d8585b7700a320f01"}
Oct 30 20:19:31.733489 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.733426    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4zrpt"]
Oct 30 20:19:31.736226 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.736208    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4zrpt"
Oct 30 20:19:31.789467 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.789444    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 20:19:31.789597 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.789574    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 20:19:31.789657 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.789580    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 20:19:31.789657 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.789619    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 20:19:31.789720 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.789649    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jxfwk\""
Oct 30 20:19:31.795315 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.795292    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4zrpt"]
Oct 30 20:19:31.827358 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.827331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/785f0704-ff0e-44e7-bc97-be771a84e03a-signing-cabundle\") pod \"service-ca-5fffc97bdf-4zrpt\" (UID: \"785f0704-ff0e-44e7-bc97-be771a84e03a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4zrpt"
Oct 30 20:19:31.827454 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.827398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/785f0704-ff0e-44e7-bc97-be771a84e03a-signing-key\") pod \"service-ca-5fffc97bdf-4zrpt\" (UID: \"785f0704-ff0e-44e7-bc97-be771a84e03a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4zrpt"
Oct 30 20:19:31.827491 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.827481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zkb7\" (UniqueName: \"kubernetes.io/projected/785f0704-ff0e-44e7-bc97-be771a84e03a-kube-api-access-4zkb7\") pod \"service-ca-5fffc97bdf-4zrpt\" (UID: \"785f0704-ff0e-44e7-bc97-be771a84e03a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4zrpt"
Oct 30 20:19:31.928663 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.928636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zkb7\" (UniqueName: \"kubernetes.io/projected/785f0704-ff0e-44e7-bc97-be771a84e03a-kube-api-access-4zkb7\") pod \"service-ca-5fffc97bdf-4zrpt\" (UID: \"785f0704-ff0e-44e7-bc97-be771a84e03a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4zrpt"
Oct 30 20:19:31.928824 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.928701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/785f0704-ff0e-44e7-bc97-be771a84e03a-signing-cabundle\") pod \"service-ca-5fffc97bdf-4zrpt\" (UID: \"785f0704-ff0e-44e7-bc97-be771a84e03a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4zrpt"
Oct 30 20:19:31.928892 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.928870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/785f0704-ff0e-44e7-bc97-be771a84e03a-signing-key\") pod \"service-ca-5fffc97bdf-4zrpt\" (UID: \"785f0704-ff0e-44e7-bc97-be771a84e03a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4zrpt"
Oct 30 20:19:31.929507 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.929404    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/785f0704-ff0e-44e7-bc97-be771a84e03a-signing-cabundle\") pod \"service-ca-5fffc97bdf-4zrpt\" (UID: \"785f0704-ff0e-44e7-bc97-be771a84e03a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4zrpt"
Oct 30 20:19:31.932047 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.932027    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/785f0704-ff0e-44e7-bc97-be771a84e03a-signing-key\") pod \"service-ca-5fffc97bdf-4zrpt\" (UID: \"785f0704-ff0e-44e7-bc97-be771a84e03a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4zrpt"
Oct 30 20:19:31.991058 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:31.990987    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zkb7\" (UniqueName: \"kubernetes.io/projected/785f0704-ff0e-44e7-bc97-be771a84e03a-kube-api-access-4zkb7\") pod \"service-ca-5fffc97bdf-4zrpt\" (UID: \"785f0704-ff0e-44e7-bc97-be771a84e03a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4zrpt"
Oct 30 20:19:32.030000 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:32.029974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:32.030152 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:32.030010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:32.030152 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:32.030035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:32.030152 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:32.030096    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 20:19:32.030152 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:32.030133    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:19:32.030152 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:32.030134    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:32.030387 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:32.030183    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls podName:4d5ab344-e8d9-4b66-8e1f-6283db623632 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:34.03016755 +0000 UTC m=+34.039515826 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls") pod "dns-default-jlxjn" (UID: "4d5ab344-e8d9-4b66-8e1f-6283db623632") : secret "dns-default-metrics-tls" not found
Oct 30 20:19:32.030387 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:32.030204    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-rbac-proxy-cm podName:98a309d0-a2a2-4ec4-82c4-5966823b6e06 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:34.030194243 +0000 UTC m=+34.039542513 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6pstp" (UID: "98a309d0-a2a2-4ec4-82c4-5966823b6e06") : configmap "kube-rbac-proxy" not found
Oct 30 20:19:32.030387 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:32.030218    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls podName:98a309d0-a2a2-4ec4-82c4-5966823b6e06 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:34.030210021 +0000 UTC m=+34.039558286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6pstp" (UID: "98a309d0-a2a2-4ec4-82c4-5966823b6e06") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:32.056091 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:32.056064    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4zrpt"
Oct 30 20:19:32.438211 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:32.438114    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4zrpt"]
Oct 30 20:19:32.440741 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:32.440709    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785f0704_ff0e_44e7_bc97_be771a84e03a.slice/crio-6dc7cc704bf02f84fdd6a8f88754657f5ccec38a7a5ca997195b469a969e153d WatchSource:0}: Error finding container 6dc7cc704bf02f84fdd6a8f88754657f5ccec38a7a5ca997195b469a969e153d: Status 404 returned error can't find the container with id 6dc7cc704bf02f84fdd6a8f88754657f5ccec38a7a5ca997195b469a969e153d
Oct 30 20:19:32.635070 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:32.635033    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4zrpt" event={"ID":"785f0704-ff0e-44e7-bc97-be771a84e03a","Type":"ContainerStarted","Data":"6dc7cc704bf02f84fdd6a8f88754657f5ccec38a7a5ca997195b469a969e153d"}
Oct 30 20:19:33.640143 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:33.640061    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-m8znf-jgfmg" event={"ID":"09961adc-4403-44ec-8c11-81938ae4fb16","Type":"ContainerStarted","Data":"5e3624625643302dc07595a1d52704a41c51446724d6b6ab46d3a3636892d3c6"}
Oct 30 20:19:33.756659 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:33.756603    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-m8znf-jgfmg" podStartSLOduration=1.402720137 podStartE2EDuration="3.756586594s" totalImagesPullingTime="2.353866457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:33.756480638 +0000 UTC m=+33.765828923" watchObservedRunningTime="2026-10-30 20:19:33.756586594 +0000 UTC m=+33.765934878"
Oct 30 20:19:34.044845 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:34.044757    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:34.044845 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:34.044805    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:34.045063 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:34.044863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:34.045063 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:34.044914    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 20:19:34.045063 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:34.044950    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:34.045063 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:34.044956    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:19:34.045063 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:34.044991    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-rbac-proxy-cm podName:98a309d0-a2a2-4ec4-82c4-5966823b6e06 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:38.044973797 +0000 UTC m=+38.054322096 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6pstp" (UID: "98a309d0-a2a2-4ec4-82c4-5966823b6e06") : configmap "kube-rbac-proxy" not found
Oct 30 20:19:34.045307 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:34.045071    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls podName:98a309d0-a2a2-4ec4-82c4-5966823b6e06 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:38.045052139 +0000 UTC m=+38.054400410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6pstp" (UID: "98a309d0-a2a2-4ec4-82c4-5966823b6e06") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:34.045307 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:34.045102    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls podName:4d5ab344-e8d9-4b66-8e1f-6283db623632 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:38.045091596 +0000 UTC m=+38.054439864 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls") pod "dns-default-jlxjn" (UID: "4d5ab344-e8d9-4b66-8e1f-6283db623632") : secret "dns-default-metrics-tls" not found
Oct 30 20:19:34.348130 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:34.348050    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl5jc\" (UniqueName: \"kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc\") pod \"network-check-target-d9xr4\" (UID: \"b9a64b7e-6436-4415-86f9-735350f24e35\") " pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:34.348130 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:34.348108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:34.348371 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:34.348180    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs\") pod \"network-metrics-daemon-vkj2j\" (UID: \"c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256\") " pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:34.348371 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:34.348264    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 20:19:34.348371 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:34.348342    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs podName:c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256 nodeName:}" failed. No retries permitted until 2026-10-30 20:20:06.348306296 +0000 UTC m=+66.357654575 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs") pod "network-metrics-daemon-vkj2j" (UID: "c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256") : secret "metrics-daemon-secret" not found
Oct 30 20:19:34.350732 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:34.350696    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:34.351156 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:34.351136    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl5jc\" (UniqueName: \"kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc\") pod \"network-check-target-d9xr4\" (UID: \"b9a64b7e-6436-4415-86f9-735350f24e35\") " pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:34.459226 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:34.459195    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:34.468196 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:34.468171    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:34.644379 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:34.644267    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mzdm7" event={"ID":"cb8a13f8-5b1b-4a9d-965f-ad3e29051484","Type":"ContainerStarted","Data":"df1cf76f1a4f8df4bd23160f113d29036391abb2c5a9f4504b4e24527fcdeca6"}
Oct 30 20:19:34.764072 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:34.764025    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mzdm7" podStartSLOduration=1.708819071 podStartE2EDuration="4.764010872s" totalImagesPullingTime="3.055191801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:34.76325123 +0000 UTC m=+34.772599516" watchObservedRunningTime="2026-10-30 20:19:34.764010872 +0000 UTC m=+34.773359147"
Oct 30 20:19:34.926752 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:34.925725    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d9xr4"]
Oct 30 20:19:34.926752 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:34.926477    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jkz4p"]
Oct 30 20:19:34.928210 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:34.928186    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf399aaa3_b634_40e9_9ca7_0941f0ec2925.slice/crio-90cfae4c28ce1b1dc06630bdf64f2c0958ec3fc6bf2f9ff17902bf12baab51bb WatchSource:0}: Error finding container 90cfae4c28ce1b1dc06630bdf64f2c0958ec3fc6bf2f9ff17902bf12baab51bb: Status 404 returned error can't find the container with id 90cfae4c28ce1b1dc06630bdf64f2c0958ec3fc6bf2f9ff17902bf12baab51bb
Oct 30 20:19:34.936117 ip-10-0-138-34 kubenswrapper[2577]: W1030 20:19:34.936095    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a64b7e_6436_4415_86f9_735350f24e35.slice/crio-7f763950851f16236c2b53ed786185db42f2642fb3c3f353c3422d14b1ba420a WatchSource:0}: Error finding container 7f763950851f16236c2b53ed786185db42f2642fb3c3f353c3422d14b1ba420a: Status 404 returned error can't find the container with id 7f763950851f16236c2b53ed786185db42f2642fb3c3f353c3422d14b1ba420a
Oct 30 20:19:35.647903 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:35.647870    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jkz4p" event={"ID":"f399aaa3-b634-40e9-9ca7-0941f0ec2925","Type":"ContainerStarted","Data":"90cfae4c28ce1b1dc06630bdf64f2c0958ec3fc6bf2f9ff17902bf12baab51bb"}
Oct 30 20:19:35.649310 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:35.649288    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4zrpt" event={"ID":"785f0704-ff0e-44e7-bc97-be771a84e03a","Type":"ContainerStarted","Data":"b66b738370de8cb11bb2ad23047975a4597975ffde50f4cd3945ac6e640c6970"}
Oct 30 20:19:35.650361 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:35.650342    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d9xr4" event={"ID":"b9a64b7e-6436-4415-86f9-735350f24e35","Type":"ContainerStarted","Data":"7f763950851f16236c2b53ed786185db42f2642fb3c3f353c3422d14b1ba420a"}
Oct 30 20:19:35.771406 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:35.771359    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-4zrpt" podStartSLOduration=2.110024147 podStartE2EDuration="4.771345365s" totalImagesPullingTime="2.661321218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:35.771183311 +0000 UTC m=+35.780531600" watchObservedRunningTime="2026-10-30 20:19:35.771345365 +0000 UTC m=+35.780693649"
Oct 30 20:19:38.080093 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:38.080060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:38.080093 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:38.080095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:38.080599 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:38.080121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:38.080599 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:38.080262    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:38.080599 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:38.080268    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:19:38.080854 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:38.080836    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls podName:98a309d0-a2a2-4ec4-82c4-5966823b6e06 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:46.08034279 +0000 UTC m=+46.089691067 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6pstp" (UID: "98a309d0-a2a2-4ec4-82c4-5966823b6e06") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:38.081128 ip-10-0-138-34 kubenswrapper[2577]: E1030 20:19:38.081102    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls podName:4d5ab344-e8d9-4b66-8e1f-6283db623632 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:46.081038217 +0000 UTC m=+46.090386481 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls") pod "dns-default-jlxjn" (UID: "4d5ab344-e8d9-4b66-8e1f-6283db623632") : secret "dns-default-metrics-tls" not found
Oct 30 20:19:38.081194 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:38.081178    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:39.663578 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:39.663537    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d9xr4" event={"ID":"b9a64b7e-6436-4415-86f9-735350f24e35","Type":"ContainerStarted","Data":"ad179b9d6b820b0c249e858a95dabdf0e92731b1a6fb118605d4a5f230c1b3a0"}
Oct 30 20:19:39.664017 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:39.663698    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:39.781678 ip-10-0-138-34 kubenswrapper[2577]: I1030 20:19:39.781630    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-d9xr4" podStartSLOduration=35.055733649 podStartE2EDuration="38.781615621s" totalImagesPullingTime="3.725881972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:39.780672703 +0000 UTC m=+39.790020999" watchObservedRunningTime="2026-10-30 20:19:39.781615621 +0000 UTC m=+39.790963919"
Oct 30 20:19:40.269335 ip-10-0-138-34 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 20:19:40.277000 ip-10-0-138-34 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 20:19:40.277163 ip-10-0-138-34 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 20:19:40.277226 ip-10-0-138-34 systemd[1]: kubelet.service: Consumed 2.274s CPU time.
Oct 30 20:19:40.291901 ip-10-0-138-34 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 20:19:40.352224 ip-10-0-138-34 kubenswrapper[4471]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:19:40.352224 ip-10-0-138-34 kubenswrapper[4471]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 20:19:40.352224 ip-10-0-138-34 kubenswrapper[4471]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:19:40.352224 ip-10-0-138-34 kubenswrapper[4471]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:19:40.354480 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354392    4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:40.354480 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354477    4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:40.354480 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354483    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:40.354480 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354486    4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:40.354747 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354490    4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:40.354747 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354494    4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:40.354747 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354498    4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:40.354747 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354502    4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:40.354747 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354505    4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:40.354747 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354509    4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:40.354747 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354512    4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:40.354747 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354516    4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:40.354747 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354519    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:40.354747 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354523    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:40.354747 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354527    4471 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:40.354747 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354531    4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:40.354747 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354535    4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:40.354747 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354539    4471 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:40.354747 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354542    4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:40.354747 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354546    4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:40.354747 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354549    4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:40.354747 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354554    4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:40.354747 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354557    4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:40.354747 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354563    4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:40.355577 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354569    4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:40.355577 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354574    4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:40.355577 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354577    4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:40.355577 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354587    4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:40.355577 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354591    4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:40.355577 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354595    4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:40.355577 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354600    4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:40.355577 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354604    4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:40.355577 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354608    4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:40.355577 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354612    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:40.355577 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354616    4471 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:40.355577 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354620    4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:40.355577 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354625    4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:40.355577 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354629    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:40.355577 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354633    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:40.355577 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354637    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:40.355577 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354641    4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:40.355577 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354645    4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:40.355577 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354649    4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:40.356369 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354653    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:40.356369 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354657    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:40.356369 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354661    4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:40.356369 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354665    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:40.356369 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354669    4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:40.356369 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354673    4471 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:40.356369 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354678    4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:40.356369 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354681    4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:40.356369 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354685    4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:40.356369 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354692    4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:40.356369 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354698    4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:40.356369 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354702    4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:40.356369 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354707    4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:40.356369 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354711    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:40.356369 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354715    4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:40.356369 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354720    4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:40.356369 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354724    4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:40.356369 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354729    4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:40.356369 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354733    4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:40.357296 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354737    4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:40.357296 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354743    4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:40.357296 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354747    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:40.357296 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354751    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:40.357296 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354755    4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:40.357296 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354760    4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:40.357296 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354764    4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:40.357296 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354768    4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:40.357296 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354772    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:40.357296 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354777    4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:40.357296 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354781    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:40.357296 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354785    4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:40.357296 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354789    4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:40.357296 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354793    4471 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:40.357296 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354797    4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:40.357296 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354802    4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:40.357296 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354806    4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:40.357296 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354810    4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:40.357296 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354815    4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:40.358136 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354818    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:40.358136 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354823    4471 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:40.358136 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354827    4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:40.358136 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354831    4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:40.358136 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354835    4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:40.358136 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354839    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:40.358136 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354843    4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:40.358136 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354847    4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:40.358136 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354851    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:40.358136 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354856    4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:40.358136 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354860    4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:40.358136 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354864    4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:40.358136 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354869    4471 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:40.358136 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354873    4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:40.358136 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354877    4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:40.358136 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354881    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:40.358136 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354885    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:40.358136 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354889    4471 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:40.358136 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354893    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:40.358136 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354898    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:40.359083 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354902    4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:40.359083 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354906    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:40.359083 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354910    4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:40.359083 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354914    4471 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:40.359083 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.354918    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:40.359083 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356159    4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:40.359083 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356170    4471 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:40.359083 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356175    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:40.359083 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356179    4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:40.359083 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356183    4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:40.359083 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356187    4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:40.359083 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356191    4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:40.359083 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356195    4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:40.359083 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356199    4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:40.359083 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356203    4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:40.359083 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356207    4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:40.359083 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356211    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:40.359083 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356216    4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:40.359083 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356220    4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:40.359083 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356224    4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:40.359990 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356229    4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:40.359990 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356233    4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:40.359990 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356237    4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:40.359990 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356241    4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:40.359990 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356245    4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:40.359990 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356250    4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:40.359990 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356255    4471 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:40.359990 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356259    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:40.359990 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356263    4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:40.359990 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356268    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:40.359990 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356272    4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:40.359990 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356276    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:40.359990 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356281    4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:40.359990 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356285    4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:40.359990 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356289    4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:40.359990 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356293    4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:40.359990 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356297    4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:40.359990 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356301    4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:40.359990 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356305    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:40.359990 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356309    4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:40.360743 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356314    4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:40.360743 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356346    4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:40.360743 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356350    4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:40.360743 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356354    4471 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:40.360743 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356359    4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:40.360743 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356363    4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:40.360743 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356367    4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:40.360743 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356371    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:40.360743 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356374    4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:40.360743 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356378    4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:40.360743 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356382    4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:40.360743 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356385    4471 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:40.360743 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356391    4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:40.360743 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356398    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:40.360743 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356403    4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:40.360743 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356407    4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:40.360743 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356411    4471 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:40.360743 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356415    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:40.360743 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356420    4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:40.361203 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356424    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:40.361203 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356428    4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:40.361203 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356432    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:40.361203 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356436    4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:40.361203 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356440    4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:40.361203 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356445    4471 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:40.361203 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356449    4471 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:40.361203 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356453    4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:40.361203 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356457    4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:40.361203 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356461    4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:40.361203 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356465    4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:40.361203 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356472    4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:40.361203 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356476    4471 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:40.361203 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356480    4471 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:40.361203 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356485    4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:40.361203 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356489    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:40.361203 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356493    4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:40.361203 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356497    4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:40.361203 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356502    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:40.361203 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356506    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:40.361699 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356509    4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:40.361699 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356513    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:40.361699 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356517    4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:40.361699 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356520    4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:40.361699 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356525    4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:40.361699 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356531    4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:40.361699 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356535    4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:40.361699 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356539    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:40.361699 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356544    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:40.361699 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356548    4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:40.361699 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356552    4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:40.361699 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356556    4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:40.361699 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356560    4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:40.361699 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356564    4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:40.361699 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356567    4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:40.361699 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356571    4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:40.361699 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356575    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:40.361699 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356579    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:40.361699 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356583    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:40.362173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356587    4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:40.362173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356590    4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:40.362173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356594    4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:40.362173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356598    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:40.362173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356601    4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:40.362173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356605    4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:40.362173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356609    4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:40.362173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356612    4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:40.362173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356624    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:40.362173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356627    4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:40.362173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356631    4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:40.362173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356635    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:40.362173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.356644    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:40.362173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357473    4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:40.362173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357482    4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:40.362173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357487    4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:40.362173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357492    4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:40.362173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357496    4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:40.362173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357500    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:40.362754 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357504    4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:40.362754 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357508    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:40.362754 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357512    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:40.362754 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357517    4471 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:40.362754 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357521    4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:40.362754 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357525    4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:40.362754 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357530    4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:40.362754 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357535    4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:40.362754 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357539    4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:40.362754 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357544    4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:40.362754 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357548    4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:40.362754 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357552    4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:40.362754 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357556    4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:40.362754 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357560    4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:40.362754 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357564    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:40.362754 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357568    4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:40.362754 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357573    4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:40.362754 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357577    4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:40.362754 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357581    4471 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:40.362754 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357585    4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:40.363222 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357589    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:40.363222 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357593    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:40.363222 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357597    4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:40.363222 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357601    4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:40.363222 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357605    4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:40.363222 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357612    4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:40.363222 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357617    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:40.363222 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357620    4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:40.363222 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357624    4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:40.363222 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357629    4471 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:40.363222 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357632    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:40.363222 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357638    4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:40.363222 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357642    4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:40.363222 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357646    4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:40.363222 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357650    4471 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:40.363222 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357654    4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:40.363222 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357659    4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:40.363222 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357663    4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:40.363222 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357667    4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:40.363222 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357675    4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:40.363719 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357680    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:40.363719 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357683    4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:40.363719 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357687    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:40.363719 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357692    4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:40.363719 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357696    4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:40.363719 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357700    4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:40.363719 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357704    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:40.363719 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357708    4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:40.363719 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357712    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:40.363719 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357716    4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:40.363719 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357719    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:40.363719 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357723    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:40.363719 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357727    4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:40.363719 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357732    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:40.363719 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357736    4471 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:40.363719 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357740    4471 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:40.363719 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357744    4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:40.363719 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357754    4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:40.363719 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357760    4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:40.364151 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357764    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:40.364151 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357768    4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:40.364151 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357782    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:40.364151 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357787    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:40.364151 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357793    4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:40.364151 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357797    4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:40.364151 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357801    4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:40.364151 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357806    4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:40.364151 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357810    4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:40.364151 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357814    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:40.364151 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357818    4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:40.364151 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357822    4471 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:40.364151 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357826    4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:40.364151 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357831    4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:40.364151 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357835    4471 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:40.364151 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357840    4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:40.364151 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357844    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:40.364151 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357848    4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:40.364151 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357854    4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:40.364151 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357859    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:40.364673 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357864    4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:40.364673 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357875    4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:40.364673 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357879    4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:40.364673 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357883    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:40.364673 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357887    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:40.364673 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357892    4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:40.364673 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357896    4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:40.364673 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357899    4471 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:40.364673 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357903    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:40.364673 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357907    4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:40.364673 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357913    4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:40.364673 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357917    4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:40.364673 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357921    4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:40.364673 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357925    4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:40.364673 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357929    4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:40.364673 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357933    4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:40.364673 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357938    4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:40.364673 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357942    4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:40.364673 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357946    4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:40.364673 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357952    4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.357956    4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358098    4471 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358109    4471 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358119    4471 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358125    4471 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358131    4471 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358137    4471 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358143    4471 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358149    4471 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358153    4471 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358157    4471 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358162    4471 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358167    4471 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358172    4471 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358177    4471 flags.go:64] FLAG: --cgroup-root=""
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358181    4471 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358186    4471 flags.go:64] FLAG: --client-ca-file=""
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358190    4471 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358195    4471 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358200    4471 flags.go:64] FLAG: --cluster-domain=""
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358204    4471 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358209    4471 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358217    4471 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 20:19:40.365159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358222    4471 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358228    4471 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358232    4471 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358237    4471 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358242    4471 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358247    4471 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358251    4471 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358256    4471 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358261    4471 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358265    4471 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358271    4471 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358276    4471 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358280    4471 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358285    4471 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358289    4471 flags.go:64] FLAG: --enable-server="true"
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358294    4471 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358302    4471 flags.go:64] FLAG: --event-burst="100"
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358306    4471 flags.go:64] FLAG: --event-qps="50"
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358311    4471 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358316    4471 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358337    4471 flags.go:64] FLAG: --eviction-hard=""
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358343    4471 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358347    4471 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358352    4471 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358356    4471 flags.go:64] FLAG: --eviction-soft=""
Oct 30 20:19:40.365886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358361    4471 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358366    4471 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358370    4471 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358374    4471 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358378    4471 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358383    4471 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358389    4471 flags.go:64] FLAG: --feature-gates=""
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358395    4471 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358400    4471 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358405    4471 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358410    4471 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358415    4471 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358420    4471 flags.go:64] FLAG: --help="false"
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358425    4471 flags.go:64] FLAG: --hostname-override="ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358430    4471 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358434    4471 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358438    4471 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358443    4471 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358449    4471 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358453    4471 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358464    4471 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358470    4471 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358475    4471 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 20:19:40.366903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358480    4471 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358487    4471 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358491    4471 flags.go:64] FLAG: --kube-reserved=""
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358496    4471 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358500    4471 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358505    4471 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358510    4471 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358514    4471 flags.go:64] FLAG: --lock-file=""
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358518    4471 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358523    4471 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358528    4471 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358536    4471 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358541    4471 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358545    4471 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358550    4471 flags.go:64] FLAG: --logging-format="text"
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358557    4471 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358563    4471 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358567    4471 flags.go:64] FLAG: --manifest-url=""
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358572    4471 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358578    4471 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358583    4471 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358588    4471 flags.go:64] FLAG: --max-pods="110"
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358593    4471 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358597    4471 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358602    4471 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 20:19:40.367722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358606    4471 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358611    4471 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358616    4471 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358620    4471 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358631    4471 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358636    4471 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358641    4471 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358645    4471 flags.go:64] FLAG: --pod-cidr=""
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358650    4471 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358655    4471 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358659    4471 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358664    4471 flags.go:64] FLAG: --port="10250"
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358668    4471 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358673    4471 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-00b0b70f5f93b92b2"
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358677    4471 flags.go:64] FLAG: --qos-reserved=""
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358682    4471 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358687    4471 flags.go:64] FLAG: --register-node="true"
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358692    4471 flags.go:64] FLAG: --register-with-taints=""
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358698    4471 flags.go:64] FLAG: --registry-burst="10"
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358702    4471 flags.go:64] FLAG: --registry-qps="5"
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358707    4471 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358711    4471 flags.go:64] FLAG: --reserved-memory=""
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358719    4471 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358724    4471 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358728    4471 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358733    4471 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 20:19:40.368416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358737    4471 flags.go:64] FLAG: --runonce="false"
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358743    4471 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358747    4471 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358752    4471 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358756    4471 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358761    4471 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358765    4471 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358770    4471 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358774    4471 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358779    4471 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358783    4471 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358788    4471 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358792    4471 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358797    4471 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358801    4471 flags.go:64] FLAG: --system-cgroups=""
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358806    4471 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358815    4471 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358819    4471 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358824    4471 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358829    4471 flags.go:64] FLAG: --tls-min-version=""
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358834    4471 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358841    4471 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358845    4471 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358850    4471 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358854    4471 flags.go:64] FLAG: --v="2"
Oct 30 20:19:40.369187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358860    4471 flags.go:64] FLAG: --version="false"
Oct 30 20:19:40.369976 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358866    4471 flags.go:64] FLAG: --vmodule=""
Oct 30 20:19:40.369976 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358872    4471 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 20:19:40.369976 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.358878    4471 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 20:19:40.369976 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359077    4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:40.369976 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359083    4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:40.369976 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359087    4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:40.369976 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359091    4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:40.369976 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359094    4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:40.369976 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359098    4471 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:40.369976 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359102    4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:40.369976 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359106    4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:40.369976 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359111    4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:40.369976 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359115    4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:40.369976 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359119    4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:40.369976 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359123    4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:40.369976 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359127    4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:40.369976 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359131    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:40.369976 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359135    4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:40.369976 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359139    4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:40.369976 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359143    4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:40.370646 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359148    4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:40.370646 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359152    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:40.370646 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359155    4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:40.370646 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359163    4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:40.370646 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359168    4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:40.370646 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359173    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:40.370646 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359178    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:40.370646 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359186    4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:40.370646 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359191    4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:40.370646 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359197    4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:40.370646 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359203    4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:40.370646 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359207    4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:40.370646 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359212    4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:40.370646 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359216    4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:40.370646 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359221    4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:40.370646 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359225    4471 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:40.370646 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359229    4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:40.370646 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359233    4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:40.370646 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359237    4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:40.371107 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359242    4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:40.371107 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359246    4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:40.371107 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359250    4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:40.371107 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359254    4471 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:40.371107 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359258    4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:40.371107 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359261    4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:40.371107 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359265    4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:40.371107 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359269    4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:40.371107 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359273    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:40.371107 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359277    4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:40.371107 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359281    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:40.371107 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359285    4471 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:40.371107 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359289    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:40.371107 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359293    4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:40.371107 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359297    4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:40.371107 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359301    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:40.371107 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359305    4471 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:40.371107 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359309    4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:40.371107 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359314    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:40.371107 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359335    4471 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:40.371599 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359342    4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:40.371599 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359346    4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:40.371599 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359350    4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:40.371599 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359354    4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:40.371599 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359358    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:40.371599 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359362    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:40.371599 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359366    4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:40.371599 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359371    4471 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:40.371599 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359376    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:40.371599 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359380    4471 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:40.371599 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359384    4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:40.371599 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359388    4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:40.371599 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359392    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:40.371599 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359396    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:40.371599 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359400    4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:40.371599 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359404    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:40.371599 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359408    4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:40.371599 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359412    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:40.371599 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359416    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:40.372089 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359420    4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:40.372089 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359425    4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:40.372089 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359429    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:40.372089 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359433    4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:40.372089 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359436    4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:40.372089 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359441    4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:40.372089 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359445    4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:40.372089 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359448    4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:40.372089 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359452    4471 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:40.372089 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359457    4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:40.372089 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359461    4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:40.372089 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359466    4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:40.372089 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359470    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:40.372089 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359477    4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:40.372089 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359481    4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:40.372089 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359485    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:40.372089 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359489    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:40.372089 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359493    4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:40.372089 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359497    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:40.372089 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359501    4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:40.372700 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359507    4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:40.372700 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359511    4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:40.372700 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359516    4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:40.372700 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359520    4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:40.372700 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359524    4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:40.372700 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359528    4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:40.372700 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359532    4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:40.372700 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359536    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:40.372700 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359540    4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:40.372700 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359544    4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:40.372700 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359548    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:40.372700 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.359590    4471 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:19:40.372700 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359644    4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:40.372700 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359651    4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:40.372700 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359656    4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:40.373078 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359660    4471 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:40.373078 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359664    4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:40.373078 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359668    4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:40.373078 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359672    4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:40.373078 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359676    4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:40.373078 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359680    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:40.373078 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359685    4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:40.373078 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359689    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:40.373078 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359693    4471 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:40.373078 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359698    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:40.373078 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359704    4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:40.373078 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359708    4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:40.373078 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359712    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:40.373078 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359716    4471 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:40.373078 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359720    4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:40.373078 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359725    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:40.373078 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359729    4471 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:40.373078 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359733    4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:40.373078 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359738    4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:40.373078 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359742    4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:40.373619 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359746    4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:40.373619 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359750    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:40.373619 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359754    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:40.373619 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359759    4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:40.373619 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359763    4471 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:40.373619 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359767    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:40.373619 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359771    4471 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:40.373619 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359776    4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:40.373619 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359780    4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:40.373619 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359784    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:40.373619 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359788    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:40.373619 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359792    4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:40.373619 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359796    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:40.373619 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359800    4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:40.373619 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359804    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:40.373619 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359808    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:40.373619 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359812    4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:40.373619 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359816    4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:40.373619 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359820    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:40.374054 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359824    4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:40.374054 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359828    4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:40.374054 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359832    4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:40.374054 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359838    4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:40.374054 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359842    4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:40.374054 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359846    4471 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:40.374054 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359851    4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:40.374054 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359855    4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:40.374054 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359859    4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:40.374054 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359863    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:40.374054 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359867    4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:40.374054 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359873    4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:40.374054 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359877    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:40.374054 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359881    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:40.374054 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359885    4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:40.374054 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359889    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:40.374054 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359893    4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:40.374054 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359898    4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:40.374054 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359902    4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:40.374054 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359906    4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:40.374559 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359910    4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:40.374559 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359914    4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:40.374559 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359918    4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:40.374559 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359922    4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:40.374559 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359926    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:40.374559 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359930    4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:40.374559 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359934    4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:40.374559 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359938    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:40.374559 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359942    4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:40.374559 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359946    4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:40.374559 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359950    4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:40.374559 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359953    4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:40.374559 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359958    4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:40.374559 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359962    4471 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:40.374559 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359966    4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:40.374559 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359971    4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:40.374559 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359976    4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:40.374559 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359981    4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:40.374559 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359985    4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:40.375023 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359989    4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:40.375023 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359993    4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:40.375023 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.359996    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:40.375023 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.360000    4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:40.375023 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.360004    4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:40.375023 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.360008    4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:40.375023 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.360012    4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:40.375023 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.360016    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:40.375023 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.360019    4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:40.375023 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.360026    4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:40.375023 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.360031    4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:40.375023 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.360036    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:40.375023 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.360040    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:40.375023 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.360044    4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:40.375023 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.360048    4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:40.375023 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.360054    4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:40.375023 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.360060    4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:40.375023 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.360064    4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:40.375023 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.360069    4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:40.375477 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.360075    4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:40.375477 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.360079    4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:40.375477 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.360085    4471 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:40.375477 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.360089    4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:40.375477 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.360093    4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:40.375477 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.360096    4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:40.375477 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.364015    4471 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 20:19:40.375477 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.364030    4471 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 20:19:40.375477 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.364047    4471 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 20:19:40.375477 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.364051    4471 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 20:19:40.375477 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364106    4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:40.375477 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364111    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:40.375477 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364115    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:40.375477 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364118    4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:40.375477 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364120    4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:40.375477 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364123    4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:40.375477 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364125    4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:40.375477 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364129    4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:40.375477 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364131    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:40.375477 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364134    4471 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:40.375938 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364136    4471 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:40.375938 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364139    4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:40.375938 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364141    4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:40.375938 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364143    4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:40.375938 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364146    4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:40.375938 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364149    4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:40.375938 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364151    4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:40.375938 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364154    4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:40.375938 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364157    4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:40.375938 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364160    4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:40.375938 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364162    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:40.375938 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364165    4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:40.375938 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364167    4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:40.375938 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364170    4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:40.375938 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364172    4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:40.375938 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364175    4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:40.375938 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364177    4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:40.375938 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364180    4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:40.375938 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364182    4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:40.375938 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364185    4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:40.376430 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364188    4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:40.376430 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364190    4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:40.376430 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364193    4471 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:40.376430 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364196    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:40.376430 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364199    4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:40.376430 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364201    4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:40.376430 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364203    4471 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:40.376430 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364206    4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:40.376430 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364208    4471 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:40.376430 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364210    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:40.376430 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364213    4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:40.376430 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364215    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:40.376430 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364217    4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:40.376430 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364220    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:40.376430 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364222    4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:40.376430 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364224    4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:40.376430 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364227    4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:40.376430 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364229    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:40.376430 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364231    4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:40.376430 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364234    4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:40.376880 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364237    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:40.376880 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364239    4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:40.376880 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364242    4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:40.376880 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364244    4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:40.376880 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364246    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:40.376880 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364249    4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:40.376880 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364251    4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:40.376880 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364254    4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:40.376880 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364256    4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:40.376880 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364259    4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:40.376880 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364261    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:40.376880 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364264    4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:40.376880 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364266    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:40.376880 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364269    4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:40.376880 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364271    4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:40.376880 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364274    4471 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:40.376880 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364276    4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:40.376880 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364279    4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:40.376880 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364281    4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:40.377396 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364284    4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:40.377396 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364287    4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:40.377396 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364289    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:40.377396 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364291    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:40.377396 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364294    4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:40.377396 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364296    4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:40.377396 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364299    4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:40.377396 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364301    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:40.377396 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364304    4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:40.377396 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364306    4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:40.377396 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364308    4471 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:40.377396 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364311    4471 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:40.377396 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364313    4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:40.377396 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364333    4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:40.377396 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364340    4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:40.377396 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364344    4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:40.377396 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364347    4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:40.377396 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364350    4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:40.377396 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364354    4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:40.377846 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364358    4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:40.377846 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364361    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:40.377846 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364364    4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:40.377846 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364367    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:40.377846 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364369    4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:40.377846 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364371    4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:40.377846 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364374    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:40.377846 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364376    4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:40.377846 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364379    4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:40.377846 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364382    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:40.377846 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364384    4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:40.377846 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364387    4471 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:40.377846 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364390    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:40.377846 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364392    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:40.377846 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364395    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:40.377846 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364397    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:40.377846 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364399    4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:40.377846 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364401    4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:40.378256 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.364427    4471 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:19:40.378256 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364555    4471 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:40.378256 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364559    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:40.378256 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364562    4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:40.378256 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364564    4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:40.378256 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364567    4471 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:40.378256 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364569    4471 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:40.378256 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364572    4471 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:40.378256 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364575    4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:40.378256 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364577    4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:40.378256 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364579    4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:40.378256 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364582    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:40.378256 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364584    4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:40.378256 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364587    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:40.378256 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364589    4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:40.378256 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364592    4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:40.378661 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364595    4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:40.378661 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364597    4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:40.378661 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364599    4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:40.378661 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364602    4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:40.378661 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364604    4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:40.378661 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364609    4471 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:40.378661 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364612    4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:40.378661 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364614    4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:40.378661 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364616    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:40.378661 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364619    4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:40.378661 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364622    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:40.378661 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364625    4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:40.378661 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364627    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:40.378661 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364630    4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:40.378661 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364632    4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:40.378661 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364635    4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:40.378661 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364637    4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:40.378661 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364640    4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:40.378661 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364643    4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:40.379100 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364647    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:40.379100 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364651    4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:40.379100 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364654    4471 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:40.379100 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364657    4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:40.379100 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364660    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:40.379100 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364662    4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:40.379100 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364664    4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:40.379100 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364667    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:40.379100 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364669    4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:40.379100 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364671    4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:40.379100 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364674    4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:40.379100 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364677    4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:40.379100 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364680    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:40.379100 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364682    4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:40.379100 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364685    4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:40.379100 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364688    4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:40.379100 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364690    4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:40.379100 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364692    4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:40.379100 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364696    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:40.379100 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364698    4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:40.379592 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364700    4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:40.379592 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364703    4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:40.379592 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364705    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:40.379592 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364707    4471 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:40.379592 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364710    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:40.379592 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364713    4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:40.379592 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364717    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:40.379592 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364719    4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:40.379592 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364722    4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:40.379592 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364724    4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:40.379592 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364727    4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:40.379592 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364729    4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:40.379592 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364732    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:40.379592 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364734    4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:40.379592 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364736    4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:40.379592 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364739    4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:40.379592 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364741    4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:40.379592 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364744    4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:40.379592 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364746    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:40.380017 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364749    4471 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:40.380017 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364751    4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:40.380017 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364753    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:40.380017 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364756    4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:40.380017 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364758    4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:40.380017 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364761    4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:40.380017 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364763    4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:40.380017 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364766    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:40.380017 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364768    4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:40.380017 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364770    4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:40.380017 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364773    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:40.380017 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364775    4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:40.380017 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364777    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:40.380017 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364780    4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:40.380017 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364782    4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:40.380017 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364784    4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:40.380017 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364787    4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:40.380017 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364789    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:40.380017 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364792    4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:40.380017 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364794    4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:40.380524 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364797    4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:40.380524 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364799    4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:40.380524 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364803    4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:40.380524 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364806    4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:40.380524 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364810    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:40.380524 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364814    4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:40.380524 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364817    4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:40.380524 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364821    4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:40.380524 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364825    4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:40.380524 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364828    4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:40.380524 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364832    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:40.380524 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364835    4471 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:40.380524 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364839    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:40.380524 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.364870    4471 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:19:40.380524 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364917    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:19:40.380524 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364922    4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:19:40.380997 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364926    4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:19:40.380997 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364930    4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:19:40.380997 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364934    4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:19:40.380997 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364937    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:19:40.380997 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364941    4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:19:40.380997 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364945    4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:19:40.380997 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364949    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:19:40.380997 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364953    4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:19:40.380997 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364957    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:19:40.380997 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364960    4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:19:40.380997 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364963    4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:19:40.380997 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364965    4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:19:40.380997 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364967    4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:19:40.380997 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364970    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:19:40.380997 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364972    4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:19:40.380997 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364975    4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:19:40.380997 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364978    4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:19:40.380997 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364980    4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:19:40.380997 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364982    4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:19:40.381723 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364985    4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:19:40.381723 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364987    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:19:40.381723 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364989    4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:19:40.381723 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364992    4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:19:40.381723 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364994    4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:19:40.381723 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364996    4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:19:40.381723 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.364999    4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:19:40.381723 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365001    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:19:40.381723 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365003    4471 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:19:40.381723 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365006    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:19:40.381723 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365008    4471 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:19:40.381723 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365011    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:19:40.381723 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365013    4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:19:40.381723 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365015    4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:19:40.381723 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365018    4471 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:19:40.381723 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365020    4471 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:19:40.381723 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365023    4471 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:19:40.381723 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365026    4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:19:40.381723 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365029    4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:19:40.381723 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365031    4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:19:40.382671 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365034    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:19:40.382671 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365036    4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:19:40.382671 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365039    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:19:40.382671 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365041    4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:19:40.382671 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365043    4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:19:40.382671 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365046    4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:19:40.382671 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365048    4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:19:40.382671 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365051    4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:19:40.382671 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365053    4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:19:40.382671 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365056    4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:19:40.382671 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365058    4471 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:19:40.382671 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365063    4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:19:40.382671 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365066    4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:19:40.382671 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365068    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:19:40.382671 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365071    4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:19:40.382671 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365073    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:19:40.382671 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365075    4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:19:40.382671 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365078    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:19:40.382671 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365080    4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:19:40.383207 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365083    4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:19:40.383207 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365085    4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:19:40.383207 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365087    4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:19:40.383207 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365090    4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:19:40.383207 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365093    4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:19:40.383207 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365096    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:19:40.383207 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365099    4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:19:40.383207 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365102    4471 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:19:40.383207 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365105    4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:19:40.383207 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365108    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:19:40.383207 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365110    4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:19:40.383207 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365113    4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:19:40.383207 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365115    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:19:40.383207 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365118    4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:19:40.383207 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365120    4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:19:40.383207 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365122    4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:19:40.383207 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365125    4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:19:40.383207 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365127    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:19:40.383207 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365129    4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:19:40.383725 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365132    4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:19:40.383725 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365134    4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:19:40.383725 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365136    4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:19:40.383725 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365139    4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:19:40.383725 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365142    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:19:40.383725 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365147    4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:19:40.383725 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365149    4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:19:40.383725 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365152    4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:19:40.383725 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365154    4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:19:40.383725 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365157    4471 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:19:40.383725 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365159    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:19:40.383725 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365163    4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:19:40.383725 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365167    4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:19:40.383725 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365170    4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:19:40.383725 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365173    4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:19:40.383725 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365176    4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:19:40.383725 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365178    4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:19:40.383725 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365180    4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:19:40.383725 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365183    4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:19:40.384173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365185    4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:19:40.384173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365188    4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:19:40.384173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365192    4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:19:40.384173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365195    4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:19:40.384173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365197    4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:19:40.384173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365201    4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:19:40.384173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365203    4471 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:19:40.384173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:40.365206    4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:19:40.384173 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.365253    4471 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 20:19:40.384173 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.367104    4471 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 20:19:40.384173 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.367180    4471 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 20:19:40.384173 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.367399    4471 server.go:1014] "Starting client certificate rotation"
Oct 30 20:19:40.384173 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.367539    4471 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:19:40.384173 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.367594    4471 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:14:01 +0000 UTC" deadline="2028-06-28 13:36:07.517114683 +0000 UTC"
Oct 30 20:19:40.384173 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.367630    4471 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14561h16m27.149487303s"
Oct 30 20:19:40.384173 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.367883    4471 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:19:40.384557 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.368926    4471 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:19:40.384557 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.370519    4471 log.go:25] "Validated CRI v1 runtime API"
Oct 30 20:19:40.384557 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.371979    4471 log.go:25] "Validated CRI v1 image API"
Oct 30 20:19:40.384557 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.372728    4471 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 20:19:40.384557 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.374117    4471 fs.go:135] Filesystem UUIDs: map[4068e041-80b4-4ea7-97b6-278deec10bd5:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9286d060-8a15-4e32-8290-6a7f0b52d327:/dev/nvme0n1p3]
Oct 30 20:19:40.384689 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.374131    4471 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/103e23ae734c5d91614d33935ae0159a9552ae90771906b38eee3ed0993b7e21/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/103e23ae734c5d91614d33935ae0159a9552ae90771906b38eee3ed0993b7e21/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/162c5e4f2c2fb66c881fa64be8b9da0cf6c2625cd4235006976d33a682598f1e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/162c5e4f2c2fb66c881fa64be8b9da0cf6c2625cd4235006976d33a682598f1e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e41ee22de1315a865116b5288b8142dfa27eefcb799f1cc10bd405a74216d6b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e41ee22de1315a865116b5288b8142dfa27eefcb799f1cc10bd405a74216d6b/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6dc7cc704bf02f84fdd6a8f88754657f5ccec38a7a5ca997195b469a969e153d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6dc7cc704bf02f84fdd6a8f88754657f5ccec38a7a5ca997195b469a969e153d/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f763950851f16236c2b53ed786185db42f2642fb3c3f353c3422d14b1ba420a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f763950851f16236c2b53ed786185db42f2642fb3c3f353c3422d14b1ba420a/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8503ccb525b56b73d936998fb4aebdf71737084bc8df057d8585b7700a320f01/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8503ccb525b56b73d936998fb4aebdf71737084bc8df057d8585b7700a320f01/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8dc8b359935965bd2c5b6908b383663e73f5b9af88074a89ceda2b99960f54ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8dc8b359935965bd2c5b6908b383663e73f5b9af88074a89ceda2b99960f54ec/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90cfae4c28ce1b1dc06630bdf64f2c0958ec3fc6bf2f9ff17902bf12baab51bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90cfae4c28ce1b1dc06630bdf64f2c0958ec3fc6bf2f9ff17902bf12baab51bb/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a24d8a5275b2394b165e3211e0c7a89cc761b741205be4feb36b8be702674f75/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a24d8a5275b2394b165e3211e0c7a89cc761b741205be4feb36b8be702674f75/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad5afd8493f2d8587877e3d0b6dacc834677ef236c98caae8deb086eb44ac13a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad5afd8493f2d8587877e3d0b6dacc834677ef236c98caae8deb086eb44ac13a/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ade061868534202ccd7cfb197607ca9672e89ef35382f8ddf9c78c1c1a9a3d4c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ade061868534202ccd7cfb197607ca9672e89ef35382f8ddf9c78c1c1a9a3d4c/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc008738dd94a51cd77630beed910b883b0a0d824345d260903c8f370fe0820d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc008738dd94a51cd77630beed910b883b0a0d824345d260903c8f370fe0820d/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc880e8168a5368cb92bfb91773c903408bdbed296f07749b8ad988036d6d243/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc880e8168a5368cb92bfb91773c903408bdbed296f07749b8ad988036d6d243/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c3ffd5d5a254a8250cac90d26fab7f14eb6aa66e8814427d6eb8aedd0861b0f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c3ffd5d5a254a8250cac90d26fab7f14eb6aa66e8814427d6eb8aedd0861b0f0/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3988a49523b442fe23502c97df5c0cd28a0c7739f7d5dcae503f258341d806e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3988a49523b442fe23502c97df5c0cd28a0c7739f7d5dcae503f258341d806e/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09961adc-4403-44ec-8c11-81938ae4fb16/volumes/kubernetes.io~projected/kube-api-access-4m5xt:{mountpoint:/var/lib/kubelet/pods/09961adc-4403-44ec-8c11-81938ae4fb16/volumes/kubernetes.io~projected/kube-api-access-4m5xt major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a0b6cf3-8303-4d98-b22f-f8a313721702/volumes/kubernetes.io~projected/kube-api-access-fxckk:{mountpoint:/var/lib/kubelet/pods/0a0b6cf3-8303-4d98-b22f-f8a313721702/volumes/kubernetes.io~projected/kube-api-access-fxckk major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1089345a-f93e-4c78-ad1c-ac38961bdf10/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/1089345a-f93e-4c78-ad1c-ac38961bdf10/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e789fe9-f262-4147-b4ee-ec42835f4ed0/volumes/kubernetes.io~projected/kube-api-access-kknl7:{mountpoint:/var/lib/kubelet/pods/3e789fe9-f262-4147-b4ee-ec42835f4ed0/volumes/kubernetes.io~projected/kube-api-access-kknl7 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d5ab344-e8d9-4b66-8e1f-6283db623632/volumes/kubernetes.io~projected/kube-api-access-7wgv9:{mountpoint:/var/lib/kubelet/pods/4d5ab344-e8d9-4b66-8e1f-6283db623632/volumes/kubernetes.io~projected/kube-api-access-7wgv9 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51727502-b532-46d0-828c-140b8eda2ba3/volumes/kubernetes.io~projected/kube-api-access-s6tv5:{mountpoint:/var/lib/kubelet/pods/51727502-b532-46d0-828c-140b8eda2ba3/volumes/kubernetes.io~projected/kube-api-access-s6tv5 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f9aa5d5-0be0-4914-a6cc-bd54151feb49/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5f9aa5d5-0be0-4914-a6cc-bd54151feb49/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f9aa5d5-0be0-4914-a6cc-bd54151feb49/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5f9aa5d5-0be0-4914-a6cc-bd54151feb49/volumes/kubernetes.io~empty-dir/tmp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f9aa5d5-0be0-4914-a6cc-bd54151feb49/volumes/kubernetes.io~projected/kube-api-access-s6s8r:{mountpoint:/var/lib/kubelet/pods/5f9aa5d5-0be0-4914-a6cc-bd54151feb49/volumes/kubernetes.io~projected/kube-api-access-s6s8r major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/785f0704-ff0e-44e7-bc97-be771a84e03a/volumes/kubernetes.io~projected/kube-api-access-4zkb7:{mountpoint:/var/lib/kubelet/pods/785f0704-ff0e-44e7-bc97-be771a84e03a/volumes/kubernetes.io~projected/kube-api-access-4zkb7 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/785f0704-ff0e-44e7-bc97-be771a84e03a/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/785f0704-ff0e-44e7-bc97-be771a84e03a/volumes/kubernetes.io~secret/signing-key major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98a309d0-a2a2-4ec4-82c4-5966823b6e06/volumes/kubernetes.io~projected/kube-api-access-ms86z:{mountpoint:/var/lib/kubelet/pods/98a309d0-a2a2-4ec4-82c4-5966823b6e06/volumes/kubernetes.io~projected/kube-api-access-ms86z major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9a64b7e-6436-4415-86f9-735350f24e35/volumes/kubernetes.io~projected/kube-api-access-vl5jc:{mountpoint:/var/lib/kubelet/pods/b9a64b7e-6436-4415-86f9-735350f24e35/volumes/kubernetes.io~projected/kube-api-access-vl5jc major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd71c9e1-948b-41cc-9d05-a65a556c8ea7/volumes/kubernetes.io~projected/kube-api-access-4c2c9:{mountpoint:/var/lib/kubelet/pods/bd71c9e1-948b-41cc-9d05-a65a556c8ea7/volumes/kubernetes.io~projected/kube-api-access-4c2c9 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256/volumes/kubernetes.io~projected/kube-api-access-vq5ft:{mountpoint:/var/lib/kubelet/pods/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256/volumes/kubernetes.io~projected/kube-api-access-vq5ft major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c85a95c6-4b03-47db-a7a3-f702a80fe01f/volumes/kubernetes.io~projected/kube-api-access-tgw7z:{mountpoint:/var/lib/kubelet/pods/c85a95c6-4b03-47db-a7a3-f702a80fe01f/volumes/kubernetes.io~projected/kube-api-access-tgw7z major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb8a13f8-5b1b-4a9d-965f-ad3e29051484/volumes/kubernetes.io~projected/kube-api-access-6nfhx:{mountpoint:/var/lib/kubelet/pods/cb8a13f8-5b1b-4a9d-965f-ad3e29051484/volumes/kubernetes.io~projected/kube-api-access-6nfhx major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb8a13f8-5b1b-4a9d-965f-ad3e29051484/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/cb8a13f8-5b1b-4a9d-965f-ad3e29051484/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f399aaa3-b634-40e9-9ca7-0941f0ec2925/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/f399aaa3-b634-40e9-9ca7-0941f0ec2925/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f82dd16e-0e8a-40ef-87bd-84560a6b9b71/volumes/kubernetes.io~projected/kube-api-access-rtjh9:{mountpoint:/var/lib/kubelet/pods/f82dd16e-0e8a-40ef-87bd-84560a6b9b71/volumes/kubernetes.io~projected/kube-api-access-rtjh9 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb857965-3140-4b4e-982c-af2680bbeff3/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/fb857965-3140-4b4e-982c-af2680bbeff3/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb857965-3140-4b4e-982c-af2680bbeff3/volumes/kubernetes.io~projected/kube-api-access-nqljx:{mountpoint:/var/lib/kubelet/pods/fb857965-3140-4b4e-982c-af2680bbeff3/volumes/kubernetes.io~projected/kube-api-access-nqljx major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb857965-3140-4b4e-982c-af2680bbeff3/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/fb857965-3140-4b4e-982c-af2680bbeff3/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a14f7155d3a0c4e9d42957e6ccd290a06dd5d5ae77f8e1df941585d2426107f9/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/523767690e1a8c2d796c6d0d57c0b534bd137fe88f4cd2c07029b68488f7f670/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/4b7ba7e0c39ae5494755abc95be7d8d30b4c7c7ccfd6d9a078356a7696d71521/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/2f3036eb2054c654d3c059c78844412797559c6c86d4eb221c1772c400b23ee4/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/f02d9fbb585b3a43f7a9b3a4d4b84e291a1afd7f2f58eaf858153b8e06645dbc/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/6cf2218e14359974a784a7c832897d409c8171efc379c4f8da6bddb218d3818d/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/8d5a86b82182b4d3a2f103665879d121e880fa9ca606459a6687bf74b3243458/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/139e1c430ce13ea0391b256e12303de4c50909e3b4f1739e44d0a2bb95214cd4/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/86320bc86c08c22b2af69c89987da97770cf441c9821429818e658e50870a452/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/8fe560fbf4335274e7b07031f22349b4e25433e1eb9263865d7bb6c9af3361b4/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/934238cf24845cdbb1a7f20270951dd16ecb9457d0aed6ebd4359917da89d0de/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/c120413922a4460ebdc686489af03d8533b1d10d99a0a831c4c0291c1a09d5c9/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/03a23f59977bfde8592dfcdd754e1509eb729e4bdf4a466312ed28c5d6dca27d/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/1c66563464aece58640f6058b9064f6030dbb74a7f3c4545241c53f67e2d9bac/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/4a6aee2b0e84dd21ccead162b61a8a68dea180af869a9f114689c4a144991afa/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/f9a287679e2f45cb2f5cdc501cfb7001b7ac77a9bfffd27841379760404bdcb1/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/700ab41c3e40e11e30743bcf700f5dc31ed242d17407d5e03ba1fd03d082952b/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/fe541b32850998f5545021b39c5b58c73160c4679e5a585ce1248496207c3937/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/f74b0d36ce3dcff5593d30b22d3bbacb4792b1e3e630a568b051b65515cb3d15/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/ef70388622fced240939ad2b40cb48302f983b49268fcbe7607f5a19a63dd7d4/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/4773bc684d3d7d9dc4a6c19938e9548de2f39eda8bfeeddc91b941244aacbda8/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-208:{mountpoint:/var/lib/containers/storage/overlay/e9d5082b7fbe904d874ee2febe54b91c7001561e6a4f54e891d97e0d908877ee/merged major:0 minor:208 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/ed36a3cd02145551d28da1247425a677eb136d0dcb542ae9c9041fefc77b7040/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/8c559b004491c868a0792b7f13f49a21f7edf6ca98d84caa4d2eff401891177c/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/6067537f5b6041005693397ca66f59715d9b70720f74b335fdc6860b59071f8a/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/79f64b4ebd7e2efbe81a5820f6ebe85b8bac0f0677090b56a29f7993bad6ffab/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/aaf2f2446a5bce661f35cea7e0bff4ac5836b6208fa68457866ad2d80b341847/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/0544835fb79afa52d1030f401eef58997bf09c3f94c700d0f9c5f29d95faf1a9/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/dda80c3aee5198a420de52541d658375221bb66c568cb250eca92f3b1e4922d9/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/b5df7e61aebdf549b681744086ae6c039589131d048e4bd8ddb160bfd671cd5d/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/5274419a410720c865aae974f3d312a20f496a48ced69cd0429b78d0d067cc1c/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/11e30d223493df42dbb314b5bfd561923f05b2b711a19917081776dc6c6977c0/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/6b849195e3654a9840c509da4f0f2330630247cb251ee2e82b18e62215bc2258/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/49331b2a69fae77de2b1d1bd496a422a9a352a8c2b7d0c68ad4b7951117b9a8f/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/1cbd09151a3dfbb0bdd08c69b8146d77e2603fe01e0f36677b77755bc2a7b307/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/d7ed8b768ced5c993a201ce9c290ebc99082a1f95e5660e2358efe9f6cbcaf38/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/770d6822884d7ba135f5511779eb8eb962ab1d4d746f80e7ce77695fcf808b37/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/193a85f5a3fb82f2cd02e461dd261ccf4853acad00d19f81b7211ec0941a63ea/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/f5db1cb2366983478c88638e568568a7f861a146f784aabb7bdd5f34e09c35bc/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/f763454fe27f79ff083bf86cb89ee5613c19fe1fc8df1a795fe11e2e3d9787c1/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/8c4477f1f09274e7cd3faa618cb33c4774fe8fc94663572f78406137ce490294/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 20:19:40.384821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.379819    4471 manager.go:217] Machine: {Timestamp:2026-10-30 20:19:40.379390142 +0000 UTC m=+0.058018747 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200860 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27fc0f28adc583cdf1717034fa200d SystemUUID:ec27fc0f-28ad-c583-cdf1-717034fa200d BootID:91fb4842-1ec5-4166-85a0-670e308d1989 Filesystems:[{Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8503ccb525b56b73d936998fb4aebdf71737084bc8df057d8585b7700a320f01/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f763950851f16236c2b53ed786185db42f2642fb3c3f353c3422d14b1ba420a/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256/volumes/kubernetes.io~projected/kube-api-access-vq5ft DeviceMajor:0 DeviceMinor:67 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c3ffd5d5a254a8250cac90d26fab7f14eb6aa66e8814427d6eb8aedd0861b0f0/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3e789fe9-f262-4147-b4ee-ec42835f4ed0/volumes/kubernetes.io~projected/kube-api-access-kknl7 DeviceMajor:0 DeviceMinor:60 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb857965-3140-4b4e-982c-af2680bbeff3/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a0b6cf3-8303-4d98-b22f-f8a313721702/volumes/kubernetes.io~projected/kube-api-access-fxckk DeviceMajor:0 DeviceMinor:58 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/5f9aa5d5-0be0-4914-a6cc-bd54151feb49/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:62 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f399aaa3-b634-40e9-9ca7-0941f0ec2925/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:226 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f82dd16e-0e8a-40ef-87bd-84560a6b9b71/volumes/kubernetes.io~projected/kube-api-access-rtjh9 DeviceMajor:0 DeviceMinor:59 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/51727502-b532-46d0-828c-140b8eda2ba3/volumes/kubernetes.io~projected/kube-api-access-s6tv5 DeviceMajor:0 DeviceMinor:63 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a24d8a5275b2394b165e3211e0c7a89cc761b741205be4feb36b8be702674f75/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc008738dd94a51cd77630beed910b883b0a0d824345d260903c8f370fe0820d/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/103e23ae734c5d91614d33935ae0159a9552ae90771906b38eee3ed0993b7e21/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e41ee22de1315a865116b5288b8142dfa27eefcb799f1cc10bd405a74216d6b/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/162c5e4f2c2fb66c881fa64be8b9da0cf6c2625cd4235006976d33a682598f1e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90cfae4c28ce1b1dc06630bdf64f2c0958ec3fc6bf2f9ff17902bf12baab51bb/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f9aa5d5-0be0-4914-a6cc-bd54151feb49/volumes/kubernetes.io~projected/kube-api-access-s6s8r DeviceMajor:0 DeviceMinor:64 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/785f0704-ff0e-44e7-bc97-be771a84e03a/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:212 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8dc8b359935965bd2c5b6908b383663e73f5b9af88074a89ceda2b99960f54ec/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/bd71c9e1-948b-41cc-9d05-a65a556c8ea7/volumes/kubernetes.io~projected/kube-api-access-4c2c9 DeviceMajor:0 DeviceMinor:66 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc880e8168a5368cb92bfb91773c903408bdbed296f07749b8ad988036d6d243/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb8a13f8-5b1b-4a9d-965f-ad3e29051484/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/fb857965-3140-4b4e-982c-af2680bbeff3/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb8a13f8-5b1b-4a9d-965f-ad3e29051484/volumes/kubernetes.io~projected/kube-api-access-6nfhx DeviceMajor:0 DeviceMinor:203 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/5f9aa5d5-0be0-4914-a6cc-bd54151feb49/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:61 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3988a49523b442fe23502c97df5c0cd28a0c7739f7d5dcae503f258341d806e/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/09961adc-4403-44ec-8c11-81938ae4fb16/volumes/kubernetes.io~projected/kube-api-access-4m5xt DeviceMajor:0 DeviceMinor:197 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ade061868534202ccd7cfb197607ca9672e89ef35382f8ddf9c78c1c1a9a3d4c/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d5ab344-e8d9-4b66-8e1f-6283db623632/volumes/kubernetes.io~projected/kube-api-access-7wgv9 DeviceMajor:0 DeviceMinor:201 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/98a309d0-a2a2-4ec4-82c4-5966823b6e06/volumes/kubernetes.io~projected/kube-api-access-ms86z DeviceMajor:0 DeviceMinor:202 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-208 DeviceMajor:0 DeviceMinor:208 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/785f0704-ff0e-44e7-bc97-be771a84e03a/volumes/kubernetes.io~projected/kube-api-access-4zkb7 DeviceMajor:0 DeviceMinor:213 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b9a64b7e-6436-4415-86f9-735350f24e35/volumes/kubernetes.io~projected/kube-api-access-vl5jc DeviceMajor:0 DeviceMinor:231 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c85a95c6-4b03-47db-a7a3-f702a80fe01f/volumes/kubernetes.io~projected/kube-api-access-tgw7z DeviceMajor:0 DeviceMinor:65 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad5afd8493f2d8587877e3d0b6dacc834677ef236c98caae8deb086eb44ac13a/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6dc7cc704bf02f84fdd6a8f88754657f5ccec38a7a5ca997195b469a969e153d/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1089345a-f93e-4c78-ad1c-ac38961bdf10/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/fb857965-3140-4b4e-982c-af2680bbeff3/volumes/kubernetes.io~projected/kube-api-access-nqljx DeviceMajor:0 DeviceMinor:57 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:6dc7cc704bf02f8 MacAddress:be:ca:2d:52:2a:69 Speed:10000 Mtu:8901} {Name:7f763950851f162 MacAddress:2a:f1:f7:b5:e0:ba Speed:10000 Mtu:8901} {Name:8503ccb525b56b7 MacAddress:d2:4b:e6:e4:39:53 Speed:10000 Mtu:8901} {Name:90cfae4c28ce1b1 MacAddress:ae:70:1e:7f:67:27 Speed:10000 Mtu:8901} {Name:ad5afd8493f2d85 MacAddress:62:ba:97:b5:3c:29 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:3f:91:8a:a2:c5 Speed:0 Mtu:9001} {Name:br-int MacAddress:aa:4e:cd:2f:66:f9 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:3f:91:8a:a2:c5 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:e6:a4:6d:39:6b:43 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:76:e6:88:22:52:60 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 20:19:40.384821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.380076    4471 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 20:19:40.384821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.380113    4471 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 20:19:40.384821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.380381    4471 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 20:19:40.384821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.380397    4471 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-34.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 20:19:40.384821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.380525    4471 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 20:19:40.384821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.380532    4471 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 20:19:40.384821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.380542    4471 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:19:40.384821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.380563    4471 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:19:40.384821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.380633    4471 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 20:19:40.384821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.380903    4471 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 20:19:40.384821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.381085    4471 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 20:19:40.384821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.381130    4471 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 20:19:40.384821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.381143    4471 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 20:19:40.384821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.381159    4471 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 20:19:40.384821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.381173    4471 kubelet.go:405] "Adding apiserver pod source"
Oct 30 20:19:40.384821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.381184    4471 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 20:19:40.384821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.382042    4471 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 20:19:40.384821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.382056    4471 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:19:40.384821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.382659    4471 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 20:19:40.385615 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.385598    4471 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 20:19:40.385649 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.385618    4471 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:19:40.385765 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.385753    4471 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 20:19:40.386009 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.385995    4471 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 20:19:40.386050 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.386039    4471 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 20:19:40.386158 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.386149    4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 20:19:40.386188 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.386167    4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 20:19:40.386188 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.386177    4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 20:19:40.386188 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.386186    4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 20:19:40.386257 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.386196    4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 20:19:40.386257 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.386205    4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 20:19:40.386257 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.386215    4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 20:19:40.386257 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.386224    4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 20:19:40.386257 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.386235    4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 20:19:40.386257 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.386244    4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 20:19:40.386431 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.386267    4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 20:19:40.386431 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.386279    4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 20:19:40.386431 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.386312    4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 20:19:40.386431 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.386338    4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 20:19:40.389552 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.389538    4471 server.go:1263] "Started kubelet"
Oct 30 20:19:40.389652 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.389587    4471 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 20:19:40.390062 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.389983    4471 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 20:19:40.390054 ip-10-0-138-34 systemd[1]: Started Kubernetes Kubelet.
Oct 30 20:19:40.390270 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.390063    4471 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 20:19:40.390482 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.390422    4471 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 20:19:40.391121 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.390838    4471 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 20:19:40.391837 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.391801    4471 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 20:19:40.391837 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.391821    4471 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.391941    4471 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:14:03 +0000 UTC" deadline="2028-08-01 11:53:08.433257578 +0000 UTC"
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.391971    4471 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15375h33m28.041289472s"
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:40.392002    4471 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-34.us-west-1.compute.internal\" not found"
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.392025    4471 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.392036    4471 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.392054    4471 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.393800    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09961adc-4403-44ec-8c11-81938ae4fb16" volumeName="kubernetes.io/configmap/09961adc-4403-44ec-8c11-81938ae4fb16-service-ca-bundle" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.393851    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09961adc-4403-44ec-8c11-81938ae4fb16" volumeName="kubernetes.io/empty-dir/09961adc-4403-44ec-8c11-81938ae4fb16-archives-path" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.393866    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a0b6cf3-8303-4d98-b22f-f8a313721702" volumeName="kubernetes.io/projected/0a0b6cf3-8303-4d98-b22f-f8a313721702-kube-api-access-fxckk" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.393881    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d5ab344-e8d9-4b66-8e1f-6283db623632" volumeName="kubernetes.io/configmap/4d5ab344-e8d9-4b66-8e1f-6283db623632-config-volume" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.393893    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51727502-b532-46d0-828c-140b8eda2ba3" volumeName="kubernetes.io/configmap/51727502-b532-46d0-828c-140b8eda2ba3-cni-binary-copy" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.393903    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51727502-b532-46d0-828c-140b8eda2ba3" volumeName="kubernetes.io/configmap/51727502-b532-46d0-828c-140b8eda2ba3-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.393913    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f9aa5d5-0be0-4914-a6cc-bd54151feb49" volumeName="kubernetes.io/empty-dir/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-tuned" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.393925    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f9aa5d5-0be0-4914-a6cc-bd54151feb49" volumeName="kubernetes.io/projected/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-kube-api-access-s6s8r" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.393938    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e789fe9-f262-4147-b4ee-ec42835f4ed0" volumeName="kubernetes.io/projected/3e789fe9-f262-4147-b4ee-ec42835f4ed0-kube-api-access-kknl7" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.393949    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c85a95c6-4b03-47db-a7a3-f702a80fe01f" volumeName="kubernetes.io/configmap/c85a95c6-4b03-47db-a7a3-f702a80fe01f-iptables-alerter-script" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.393960    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1089345a-f93e-4c78-ad1c-ac38961bdf10" volumeName="kubernetes.io/secret/1089345a-f93e-4c78-ad1c-ac38961bdf10-agent-certs" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.393970    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d5ab344-e8d9-4b66-8e1f-6283db623632" volumeName="kubernetes.io/projected/4d5ab344-e8d9-4b66-8e1f-6283db623632-kube-api-access-7wgv9" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.393984    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9a64b7e-6436-4415-86f9-735350f24e35" volumeName="kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.393996    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb8a13f8-5b1b-4a9d-965f-ad3e29051484" volumeName="kubernetes.io/secret/cb8a13f8-5b1b-4a9d-965f-ad3e29051484-cert" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394019    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1089345a-f93e-4c78-ad1c-ac38961bdf10" volumeName="kubernetes.io/configmap/1089345a-f93e-4c78-ad1c-ac38961bdf10-konnectivity-ca" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394029    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51727502-b532-46d0-828c-140b8eda2ba3" volumeName="kubernetes.io/projected/51727502-b532-46d0-828c-140b8eda2ba3-kube-api-access-s6tv5" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394039    4471 factory.go:55] Registering systemd factory
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394076    4471 factory.go:223] Registration of the systemd container factory successfully
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394040    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="785f0704-ff0e-44e7-bc97-be771a84e03a" volumeName="kubernetes.io/projected/785f0704-ff0e-44e7-bc97-be771a84e03a-kube-api-access-4zkb7" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394125    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98a309d0-a2a2-4ec4-82c4-5966823b6e06" volumeName="kubernetes.io/configmap/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394136    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98a309d0-a2a2-4ec4-82c4-5966823b6e06" volumeName="kubernetes.io/empty-dir/98a309d0-a2a2-4ec4-82c4-5966823b6e06-data-volume" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394148    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f399aaa3-b634-40e9-9ca7-0941f0ec2925" volumeName="kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394173    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f82dd16e-0e8a-40ef-87bd-84560a6b9b71" volumeName="kubernetes.io/configmap/f82dd16e-0e8a-40ef-87bd-84560a6b9b71-serviceca" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394184    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb857965-3140-4b4e-982c-af2680bbeff3" volumeName="kubernetes.io/configmap/fb857965-3140-4b4e-982c-af2680bbeff3-ovnkube-config" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394196    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09961adc-4403-44ec-8c11-81938ae4fb16" volumeName="kubernetes.io/configmap/09961adc-4403-44ec-8c11-81938ae4fb16-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394207    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09961adc-4403-44ec-8c11-81938ae4fb16" volumeName="kubernetes.io/projected/09961adc-4403-44ec-8c11-81938ae4fb16-kube-api-access-4m5xt" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394218    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51727502-b532-46d0-828c-140b8eda2ba3" volumeName="kubernetes.io/configmap/51727502-b532-46d0-828c-140b8eda2ba3-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394237    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c85a95c6-4b03-47db-a7a3-f702a80fe01f" volumeName="kubernetes.io/projected/c85a95c6-4b03-47db-a7a3-f702a80fe01f-kube-api-access-tgw7z" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394249    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb8a13f8-5b1b-4a9d-965f-ad3e29051484" volumeName="kubernetes.io/projected/cb8a13f8-5b1b-4a9d-965f-ad3e29051484-kube-api-access-6nfhx" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394260    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f82dd16e-0e8a-40ef-87bd-84560a6b9b71" volumeName="kubernetes.io/projected/f82dd16e-0e8a-40ef-87bd-84560a6b9b71-kube-api-access-rtjh9" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394276    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a0b6cf3-8303-4d98-b22f-f8a313721702" volumeName="kubernetes.io/configmap/0a0b6cf3-8303-4d98-b22f-f8a313721702-cni-binary-copy" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394467    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f9aa5d5-0be0-4914-a6cc-bd54151feb49" volumeName="kubernetes.io/empty-dir/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-tmp" seLinuxMountContext=""
Oct 30 20:19:40.394467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394490    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98a309d0-a2a2-4ec4-82c4-5966823b6e06" volumeName="kubernetes.io/projected/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-api-access-ms86z" seLinuxMountContext=""
Oct 30 20:19:40.396286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394516    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd71c9e1-948b-41cc-9d05-a65a556c8ea7" volumeName="kubernetes.io/empty-dir/bd71c9e1-948b-41cc-9d05-a65a556c8ea7-tmp-dir" seLinuxMountContext=""
Oct 30 20:19:40.396286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394524    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb857965-3140-4b4e-982c-af2680bbeff3" volumeName="kubernetes.io/secret/fb857965-3140-4b4e-982c-af2680bbeff3-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 20:19:40.396286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394535    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d5ab344-e8d9-4b66-8e1f-6283db623632" volumeName="kubernetes.io/empty-dir/4d5ab344-e8d9-4b66-8e1f-6283db623632-tmp-dir" seLinuxMountContext=""
Oct 30 20:19:40.396286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394543    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="785f0704-ff0e-44e7-bc97-be771a84e03a" volumeName="kubernetes.io/configmap/785f0704-ff0e-44e7-bc97-be771a84e03a-signing-cabundle" seLinuxMountContext=""
Oct 30 20:19:40.396286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394551    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="785f0704-ff0e-44e7-bc97-be771a84e03a" volumeName="kubernetes.io/secret/785f0704-ff0e-44e7-bc97-be771a84e03a-signing-key" seLinuxMountContext=""
Oct 30 20:19:40.396286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394561    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd71c9e1-948b-41cc-9d05-a65a556c8ea7" volumeName="kubernetes.io/projected/bd71c9e1-948b-41cc-9d05-a65a556c8ea7-kube-api-access-4c2c9" seLinuxMountContext=""
Oct 30 20:19:40.396286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394642    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256" volumeName="kubernetes.io/projected/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-kube-api-access-vq5ft" seLinuxMountContext=""
Oct 30 20:19:40.396286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394652    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb857965-3140-4b4e-982c-af2680bbeff3" volumeName="kubernetes.io/configmap/fb857965-3140-4b4e-982c-af2680bbeff3-env-overrides" seLinuxMountContext=""
Oct 30 20:19:40.396286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394659    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb857965-3140-4b4e-982c-af2680bbeff3" volumeName="kubernetes.io/projected/fb857965-3140-4b4e-982c-af2680bbeff3-kube-api-access-nqljx" seLinuxMountContext=""
Oct 30 20:19:40.396286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394667    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a0b6cf3-8303-4d98-b22f-f8a313721702" volumeName="kubernetes.io/configmap/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-daemon-config" seLinuxMountContext=""
Oct 30 20:19:40.396286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394677    4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb857965-3140-4b4e-982c-af2680bbeff3" volumeName="kubernetes.io/configmap/fb857965-3140-4b4e-982c-af2680bbeff3-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 20:19:40.396286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394684    4471 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 20:19:40.396286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394689    4471 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 20:19:40.396286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394954    4471 factory.go:153] Registering CRI-O factory
Oct 30 20:19:40.396286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.394969    4471 factory.go:223] Registration of the crio container factory successfully
Oct 30 20:19:40.396286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.395020    4471 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 20:19:40.396286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.395039    4471 factory.go:103] Registering Raw factory
Oct 30 20:19:40.396286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.395049    4471 manager.go:1199] Started watching for new ooms in manager
Oct 30 20:19:40.396286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.395275    4471 manager.go:319] Starting recovery of all containers
Oct 30 20:19:40.422535 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.422502    4471 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 20:19:40.423821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.423803    4471 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 20:19:40.423929 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.423835    4471 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 20:19:40.423929 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.423853    4471 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 20:19:40.423929 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:40.423886    4471 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 20:19:40.435825 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.435802    4471 generic.go:356] "Generic (PLEG): container finished" podID="be68dc42ad591cdc0bc8a690a67f92fa" containerID="6b9eddcb6f4d3460656b0c47cbe1eb4c9b4a9c9c11928de2c64a93041e6cc070" exitCode=0
Oct 30 20:19:40.443220 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.443195    4471 generic.go:356] "Generic (PLEG): container finished" podID="51727502-b532-46d0-828c-140b8eda2ba3" containerID="29e122e5925d2c60fa39954ab631634008f021c9900278c275bcb2dc1cfe7846" exitCode=0
Oct 30 20:19:40.443220 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.443218    4471 generic.go:356] "Generic (PLEG): container finished" podID="51727502-b532-46d0-828c-140b8eda2ba3" containerID="eb4e0f901d3c9e537e39e21b6df47da7aa2b4ec3f095981bde0af8f118fbb494" exitCode=0
Oct 30 20:19:40.443389 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.443226    4471 generic.go:356] "Generic (PLEG): container finished" podID="51727502-b532-46d0-828c-140b8eda2ba3" containerID="f60c093d4f453449d33b82097be4951c215229dacb03cafde45ec40297f2f6d3" exitCode=0
Oct 30 20:19:40.443389 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.443237    4471 generic.go:356] "Generic (PLEG): container finished" podID="51727502-b532-46d0-828c-140b8eda2ba3" containerID="8741dd37a60df0706d2c734dee0bc3d2957a50825e45a6e73f9154086ab991c8" exitCode=0
Oct 30 20:19:40.443389 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.443245    4471 generic.go:356] "Generic (PLEG): container finished" podID="51727502-b532-46d0-828c-140b8eda2ba3" containerID="cccb4b8a44f0dd3a1665d9ca372611bf562f31f0ea10d178af06e110bfc66f25" exitCode=0
Oct 30 20:19:40.443389 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.443253    4471 generic.go:356] "Generic (PLEG): container finished" podID="51727502-b532-46d0-828c-140b8eda2ba3" containerID="fddd934d827a804b7782856ffbb44761ae631b71df69bbf5e3ccb22897b1e0c4" exitCode=0
Oct 30 20:19:40.465293 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.465276    4471 manager.go:324] Recovery completed
Oct 30 20:19:40.477303 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.477286    4471 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:40.478406 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.478390    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:40.478478 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.478421    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:40.478478 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.478431    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:40.479791 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.479777    4471 cpu_manager.go:226] "Starting" policy="none"
Oct 30 20:19:40.479839 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.479791    4471 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 20:19:40.479839 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.479801    4471 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:40.479839 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.479810    4471 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 20:19:40.479966 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.479951    4471 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 20:19:40.479994 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.479967    4471 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 20:19:40.479994 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.479984    4471 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 20:19:40.479994 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.479990    4471 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 20:19:40.479994 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.479995    4471 policy_none.go:50] "Start"
Oct 30 20:19:40.480096 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.480004    4471 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 20:19:40.480096 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.480013    4471 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 20:19:40.480677 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.480661    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:40.480735 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.480694    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:40.480735 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.480710    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:40.481358 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.481345    4471 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 20:19:40.481389 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.481361    4471 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 20:19:40.481389 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.481368    4471 policy_none.go:44] "Start"
Oct 30 20:19:40.484940 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.484918    4471 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 20:19:40.487862 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.487843    4471 manager.go:341] "Starting Device Plugin manager"
Oct 30 20:19:40.487987 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:40.487887    4471 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 20:19:40.487987 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.487899    4471 server.go:87] "Starting device plugin registration server"
Oct 30 20:19:40.488189 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.488177    4471 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 20:19:40.488225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.488191    4471 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 20:19:40.488302 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.488287    4471 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 20:19:40.489702 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.489645    4471 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 20:19:40.489801 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.489711    4471 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 20:19:40.489801 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.489719    4471 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 20:19:40.489899 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:40.489806    4471 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 20:19:40.489899 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.489788    4471 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:19:40.489699726Z","UUID":"624cd73a-d515-42ce-8db6-38c0bd1795e8","Handler":null,"Name":"","Endpoint":""}
Oct 30 20:19:40.489899 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:40.489869    4471 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-34.us-west-1.compute.internal\" not found"
Oct 30 20:19:40.491467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.491448    4471 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 20:19:40.491558 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.491475    4471 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 20:19:40.524466 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.523988    4471 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal"]
Oct 30 20:19:40.524576 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.524548    4471 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:40.525466 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.525451    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:40.525526 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.525479    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:40.525526 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.525489    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:40.525709 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.525700    4471 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:40.525837 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.525822    4471 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:40.526539 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.526521    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:40.526626 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.526543    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:40.526626 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.526552    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:40.526626 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.526523    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:40.526768 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.526631    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:40.526768 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.526649    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:40.526768 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:40.526694    4471 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-34.us-west-1.compute.internal\" not found" node="ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:40.526957 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.526926    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal" event={"ID":"be68dc42ad591cdc0bc8a690a67f92fa","Type":"ContainerStarted","Data":"3b9109434d6aacb41b5881ddf431803ffcf6fbada1ba286a6d45856a8de1f133"}
Oct 30 20:19:40.527016 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.526968    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal" event={"ID":"be68dc42ad591cdc0bc8a690a67f92fa","Type":"ContainerDied","Data":"6b9eddcb6f4d3460656b0c47cbe1eb4c9b4a9c9c11928de2c64a93041e6cc070"}
Oct 30 20:19:40.527016 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.526991    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal" event={"ID":"be68dc42ad591cdc0bc8a690a67f92fa","Type":"ContainerStarted","Data":"c3ffd5d5a254a8250cac90d26fab7f14eb6aa66e8814427d6eb8aedd0861b0f0"}
Oct 30 20:19:40.527016 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.527009    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal" event={"ID":"d10e19b03ae8d13f3ef8ccaa1c90976f","Type":"ContainerStarted","Data":"0fa413ed30968268f16a6156e0da49568bdcba923c52aa59fdd97f8beb36da89"}
Oct 30 20:19:40.527135 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.527016    4471 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:40.527135 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.527026    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal" event={"ID":"d10e19b03ae8d13f3ef8ccaa1c90976f","Type":"ContainerStarted","Data":"8dc8b359935965bd2c5b6908b383663e73f5b9af88074a89ceda2b99960f54ec"}
Oct 30 20:19:40.527759 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.527742    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:40.527820 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.527771    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:40.527820 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.527784    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:40.527904 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:40.527894    4471 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-34.us-west-1.compute.internal\" not found" node="ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:40.587823 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.587771    4471 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:19:40.588021 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.588007    4471 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 20:19:40.588866 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.588856    4471 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:19:40.589191 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.589176    4471 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:19:40.589566 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.589550    4471 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:19:40.589842 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.589827    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:19:40.589900 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.589857    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:19:40.589900 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.589873    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:19:40.589900 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.589901    4471 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:40.593077 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.593066    4471 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:40.639177 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.639161    4471 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:19:40.685689 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.685673    4471 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:19:40.685764 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:40.685715    4471 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:40.685764 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.685730    4471 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:40.688831 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.688817    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be68dc42ad591cdc0bc8a690a67f92fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal\" (UID: \"be68dc42ad591cdc0bc8a690a67f92fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:40.688881 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.688840    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be68dc42ad591cdc0bc8a690a67f92fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal\" (UID: \"be68dc42ad591cdc0bc8a690a67f92fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:40.688881 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.688861    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d10e19b03ae8d13f3ef8ccaa1c90976f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal\" (UID: \"d10e19b03ae8d13f3ef8ccaa1c90976f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:40.736917 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.736899    4471 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:40.736981 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.736948    4471 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:40.737414 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.737399    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeNotReady"
Oct 30 20:19:40.737498 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.737430    4471 setters.go:546] "Node became not ready" node="ip-10-0-138-34.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T20:19:40Z","lastTransitionTime":"2026-10-30T20:19:40Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 30 20:19:40.740558 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.740543    4471 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:19:40.740624 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:40.740580    4471 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:40.753840 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.753818    4471 nodeinfomanager.go:471] existing CSINode "ip-10-0-138-34.us-west-1.compute.internal" is owned by different node (oldNodeID="a6835c27-10f7-46e8-9d94-a644a1bd9ca0", newNodeID=""), cleaning up...
Oct 30 20:19:40.789653 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.789635    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be68dc42ad591cdc0bc8a690a67f92fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal\" (UID: \"be68dc42ad591cdc0bc8a690a67f92fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:40.789727 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.789661    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d10e19b03ae8d13f3ef8ccaa1c90976f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal\" (UID: \"d10e19b03ae8d13f3ef8ccaa1c90976f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:40.789727 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.789678    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be68dc42ad591cdc0bc8a690a67f92fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal\" (UID: \"be68dc42ad591cdc0bc8a690a67f92fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:40.789727 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.789717    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/be68dc42ad591cdc0bc8a690a67f92fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal\" (UID: \"be68dc42ad591cdc0bc8a690a67f92fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:40.789813 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.789747    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be68dc42ad591cdc0bc8a690a67f92fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal\" (UID: \"be68dc42ad591cdc0bc8a690a67f92fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:40.789813 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:40.789755    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d10e19b03ae8d13f3ef8ccaa1c90976f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal\" (UID: \"d10e19b03ae8d13f3ef8ccaa1c90976f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:41.383103 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.383067    4471 apiserver.go:52] "Watching apiserver"
Oct 30 20:19:41.459871 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.459847    4471 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:41.460034 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.460020    4471 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:41.511127 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.511104    4471 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 20:19:41.511530 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.511512    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-vkj2j","kube-system/konnectivity-agent-qxwhp","openshift-service-ca/service-ca-5fffc97bdf-4zrpt","kube-system/kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal","openshift-multus/multus-psqxp","openshift-network-operator/iptables-alerter-r7mrf","openshift-network-diagnostics/network-check-target-d9xr4","openshift-ingress-canary/ingress-canary-mzdm7","kube-system/global-pull-secret-syncer-jkz4p","openshift-cluster-node-tuning-operator/tuned-cwm29","openshift-insights/insights-runtime-extractor-6pstp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg","openshift-ovn-kubernetes/ovnkube-node-frx59","openshift-insights/periodic-gathering-m8znf-jgfmg","openshift-dns/dns-default-jlxjn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal","openshift-dns/node-resolver-jbb5l","openshift-image-registry/node-ca-n8gtv","openshift-multus/multus-additional-cni-plugins-2tgdb"]
Oct 30 20:19:41.516834 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.516814    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:41.517935 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.517914    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:41.518295 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.518280    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:41.534403 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.534383    4471 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:19:41.534403 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.534385    4471 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:19:41.534572 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:41.534439    4471 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:41.534572 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:41.534439    4471 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-34.us-west-1.compute.internal"
Oct 30 20:19:41.563939 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.563920    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 20:19:41.564013 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.563961    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wz6nt\""
Oct 30 20:19:41.564792 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.564775    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 20:19:41.565543 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.565529    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 20:19:41.566159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.566144    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4gwdx\""
Oct 30 20:19:41.566225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.566175    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 20:19:41.566225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.566194    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kv88f\""
Oct 30 20:19:41.566645 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.566629    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 20:19:41.566722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.566644    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 20:19:41.566722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.566633    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 20:19:41.567247 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.567231    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 20:19:41.567413 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.567397    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 20:19:41.567484 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.567439    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 20:19:41.567707 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.567695    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 20:19:41.567881 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.567869    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:19:41.568262 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.568227    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 20:19:41.568262 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.568243    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:19:41.568262 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.568227    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2zwff\""
Oct 30 20:19:41.568475 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.568282    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 20:19:41.568515 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.568477    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 20:19:41.568515 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.568483    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 20:19:41.568515 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.568483    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 20:19:41.568618 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.568513    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 20:19:41.568618 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.568523    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gbmn9\""
Oct 30 20:19:41.568769 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.568751    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 20:19:41.568865 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.568836    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-k74qc\""
Oct 30 20:19:41.568939 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.568928    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vn2fz\""
Oct 30 20:19:41.569007 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.568997    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6wxmv\""
Oct 30 20:19:41.569048 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.569006    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 20:19:41.569090 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.569058    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 20:19:41.569140 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.569115    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 20:19:41.569202 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.569189    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cxq5f\""
Oct 30 20:19:41.569244 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.569217    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 20:19:41.569274 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.569253    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 20:19:41.569309 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.569288    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nnssb\""
Oct 30 20:19:41.569545 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.569522    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 20:19:41.569585 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.569547    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 20:19:41.569585 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.569571    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lm5vf\""
Oct 30 20:19:41.569670 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.569557    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 20:19:41.569670 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.569593    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 20:19:41.569670 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.569596    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5x4ch\""
Oct 30 20:19:41.584084 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.584064    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 20:19:41.584084 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.584072    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 20:19:41.584485 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.584471    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f4ct2\""
Oct 30 20:19:41.593259 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.593244    4471 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 20:19:41.594667 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.594653    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6s8r\" (UniqueName: \"kubernetes.io/projected/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-kube-api-access-s6s8r\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.594712 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.594678    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:41.594712 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.594694    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-system-cni-dir\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.594712 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.594710    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-run-netns\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.594841 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.594724    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-run-openvswitch\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.594922 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.594905    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f399aaa3-b634-40e9-9ca7-0941f0ec2925-kubelet-config\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:41.594972 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.594933    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-tuned\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.594972 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.594959    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-cnibin\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.595046 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595003    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5ab344-e8d9-4b66-8e1f-6283db623632-tmp-dir\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:41.595097 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595056    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f399aaa3-b634-40e9-9ca7-0941f0ec2925-dbus\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:41.595097 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595064    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-tuned\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.595097 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595079    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:41.595222 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595125    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5ab344-e8d9-4b66-8e1f-6283db623632-tmp-dir\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:41.595222 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595127    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51727502-b532-46d0-828c-140b8eda2ba3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:41.595222 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595187    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-hostroot\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.595415 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595229    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-run-ovn\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.595415 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595252    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-cni-bin\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.595415 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595269    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-cni-netd\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.595415 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595282    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:41.595415 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595309    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-var-lib-cni-multus\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.595415 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595367    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-var-lib-kubelet\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.595415 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595392    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-run-systemd\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.595672 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595425    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-kubernetes\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.595672 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595434    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f399aaa3-b634-40e9-9ca7-0941f0ec2925-original-pull-secret\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:41.595672 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595450    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-host\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.595672 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595450    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/51727502-b532-46d0-828c-140b8eda2ba3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:41.595672 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595473    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98a309d0-a2a2-4ec4-82c4-5966823b6e06-crio-socket\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:41.595672 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595498    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-slash\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.595672 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595516    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqljx\" (UniqueName: \"kubernetes.io/projected/fb857965-3140-4b4e-982c-af2680bbeff3-kube-api-access-nqljx\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.595672 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595557    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51727502-b532-46d0-828c-140b8eda2ba3-system-cni-dir\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:41.595672 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595585    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51727502-b532-46d0-828c-140b8eda2ba3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:41.595672 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595606    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6tv5\" (UniqueName: \"kubernetes.io/projected/51727502-b532-46d0-828c-140b8eda2ba3-kube-api-access-s6tv5\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:41.595672 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595653    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq5ft\" (UniqueName: \"kubernetes.io/projected/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-kube-api-access-vq5ft\") pod \"network-metrics-daemon-vkj2j\" (UID: \"c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256\") " pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:41.596144 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595694    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms86z\" (UniqueName: \"kubernetes.io/projected/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-api-access-ms86z\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:41.596144 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595718    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb8a13f8-5b1b-4a9d-965f-ad3e29051484-cert\") pod \"ingress-canary-mzdm7\" (UID: \"cb8a13f8-5b1b-4a9d-965f-ad3e29051484\") " pod="openshift-ingress-canary/ingress-canary-mzdm7"
Oct 30 20:19:41.596144 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595733    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f82dd16e-0e8a-40ef-87bd-84560a6b9b71-host\") pod \"node-ca-n8gtv\" (UID: \"f82dd16e-0e8a-40ef-87bd-84560a6b9b71\") " pod="openshift-image-registry/node-ca-n8gtv"
Oct 30 20:19:41.596144 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595748    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb857965-3140-4b4e-982c-af2680bbeff3-ovnkube-config\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.596144 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595763    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51727502-b532-46d0-828c-140b8eda2ba3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:41.596144 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595793    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09961adc-4403-44ec-8c11-81938ae4fb16-trusted-ca-bundle\") pod \"periodic-gathering-m8znf-jgfmg\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") " pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:19:41.596144 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595814    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-modprobe-d\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.596144 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595831    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-etc-openvswitch\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.596144 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595849    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb857965-3140-4b4e-982c-af2680bbeff3-ovnkube-script-lib\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.596144 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595872    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-cni-dir\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.596144 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595911    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd71c9e1-948b-41cc-9d05-a65a556c8ea7-tmp-dir\") pod \"node-resolver-jbb5l\" (UID: \"bd71c9e1-948b-41cc-9d05-a65a556c8ea7\") " pod="openshift-dns/node-resolver-jbb5l"
Oct 30 20:19:41.596144 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595955    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c2c9\" (UniqueName: \"kubernetes.io/projected/bd71c9e1-948b-41cc-9d05-a65a556c8ea7-kube-api-access-4c2c9\") pod \"node-resolver-jbb5l\" (UID: \"bd71c9e1-948b-41cc-9d05-a65a556c8ea7\") " pod="openshift-dns/node-resolver-jbb5l"
Oct 30 20:19:41.596144 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595960    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/51727502-b532-46d0-828c-140b8eda2ba3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:41.596144 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595979    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb857965-3140-4b4e-982c-af2680bbeff3-ovnkube-config\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.596144 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.595996    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd71c9e1-948b-41cc-9d05-a65a556c8ea7-tmp-dir\") pod \"node-resolver-jbb5l\" (UID: \"bd71c9e1-948b-41cc-9d05-a65a556c8ea7\") " pod="openshift-dns/node-resolver-jbb5l"
Oct 30 20:19:41.596144 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596003    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1089345a-f93e-4c78-ad1c-ac38961bdf10-konnectivity-ca\") pod \"konnectivity-agent-qxwhp\" (UID: \"1089345a-f93e-4c78-ad1c-ac38961bdf10\") " pod="kube-system/konnectivity-agent-qxwhp"
Oct 30 20:19:41.596144 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596085    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:41.596144 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596150    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-socket-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:41.596861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596193    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-run-k8s-cni-cncf-io\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.596861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596226    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-run-multus-certs\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.596861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596235    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1089345a-f93e-4c78-ad1c-ac38961bdf10-konnectivity-ca\") pod \"konnectivity-agent-qxwhp\" (UID: \"1089345a-f93e-4c78-ad1c-ac38961bdf10\") " pod="kube-system/konnectivity-agent-qxwhp"
Oct 30 20:19:41.596861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596253    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-etc-kubernetes\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.596861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596268    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c85a95c6-4b03-47db-a7a3-f702a80fe01f-host-slash\") pod \"iptables-alerter-r7mrf\" (UID: \"c85a95c6-4b03-47db-a7a3-f702a80fe01f\") " pod="openshift-network-operator/iptables-alerter-r7mrf"
Oct 30 20:19:41.596861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596300    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgw7z\" (UniqueName: \"kubernetes.io/projected/c85a95c6-4b03-47db-a7a3-f702a80fe01f-kube-api-access-tgw7z\") pod \"iptables-alerter-r7mrf\" (UID: \"c85a95c6-4b03-47db-a7a3-f702a80fe01f\") " pod="openshift-network-operator/iptables-alerter-r7mrf"
Oct 30 20:19:41.596861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596331    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-var-lib-openvswitch\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.596861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596349    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-log-socket\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.596861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596365    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.596861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596381    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1089345a-f93e-4c78-ad1c-ac38961bdf10-agent-certs\") pod \"konnectivity-agent-qxwhp\" (UID: \"1089345a-f93e-4c78-ad1c-ac38961bdf10\") " pod="kube-system/konnectivity-agent-qxwhp"
Oct 30 20:19:41.596861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596396    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/785f0704-ff0e-44e7-bc97-be771a84e03a-signing-key\") pod \"service-ca-5fffc97bdf-4zrpt\" (UID: \"785f0704-ff0e-44e7-bc97-be771a84e03a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4zrpt"
Oct 30 20:19:41.596861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596410    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zkb7\" (UniqueName: \"kubernetes.io/projected/785f0704-ff0e-44e7-bc97-be771a84e03a-kube-api-access-4zkb7\") pod \"service-ca-5fffc97bdf-4zrpt\" (UID: \"785f0704-ff0e-44e7-bc97-be771a84e03a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4zrpt"
Oct 30 20:19:41.596861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596448    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51727502-b532-46d0-828c-140b8eda2ba3-cni-binary-copy\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:41.596861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596513    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-etc-selinux\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:41.596861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596681    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-sys-fs\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:41.596861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596689    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1089345a-f93e-4c78-ad1c-ac38961bdf10-agent-certs\") pod \"konnectivity-agent-qxwhp\" (UID: \"1089345a-f93e-4c78-ad1c-ac38961bdf10\") " pod="kube-system/konnectivity-agent-qxwhp"
Oct 30 20:19:41.596861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596715    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09961adc-4403-44ec-8c11-81938ae4fb16-service-ca-bundle\") pod \"periodic-gathering-m8znf-jgfmg\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") " pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:19:41.596861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596749    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-sysconfig\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.596861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596777    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f82dd16e-0e8a-40ef-87bd-84560a6b9b71-serviceca\") pod \"node-ca-n8gtv\" (UID: \"f82dd16e-0e8a-40ef-87bd-84560a6b9b71\") " pod="openshift-image-registry/node-ca-n8gtv"
Oct 30 20:19:41.596861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596819    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-registration-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:41.596861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596849    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kknl7\" (UniqueName: \"kubernetes.io/projected/3e789fe9-f262-4147-b4ee-ec42835f4ed0-kube-api-access-kknl7\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:41.596861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596871    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-lib-modules\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.596969    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-var-lib-kubelet\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597000    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98a309d0-a2a2-4ec4-82c4-5966823b6e06-data-volume\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597035    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nfhx\" (UniqueName: \"kubernetes.io/projected/cb8a13f8-5b1b-4a9d-965f-ad3e29051484-kube-api-access-6nfhx\") pod \"ingress-canary-mzdm7\" (UID: \"cb8a13f8-5b1b-4a9d-965f-ad3e29051484\") " pod="openshift-ingress-canary/ingress-canary-mzdm7"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597065    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98a309d0-a2a2-4ec4-82c4-5966823b6e06-data-volume\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597072    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51727502-b532-46d0-828c-140b8eda2ba3-cni-binary-copy\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597103    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-daemon-config\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597131    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c85a95c6-4b03-47db-a7a3-f702a80fe01f-iptables-alerter-script\") pod \"iptables-alerter-r7mrf\" (UID: \"c85a95c6-4b03-47db-a7a3-f702a80fe01f\") " pod="openshift-network-operator/iptables-alerter-r7mrf"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597155    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bd71c9e1-948b-41cc-9d05-a65a556c8ea7-hosts-file\") pod \"node-resolver-jbb5l\" (UID: \"bd71c9e1-948b-41cc-9d05-a65a556c8ea7\") " pod="openshift-dns/node-resolver-jbb5l"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597179    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-systemd-units\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597203    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb857965-3140-4b4e-982c-af2680bbeff3-env-overrides\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597226    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wgv9\" (UniqueName: \"kubernetes.io/projected/4d5ab344-e8d9-4b66-8e1f-6283db623632-kube-api-access-7wgv9\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597312    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c85a95c6-4b03-47db-a7a3-f702a80fe01f-iptables-alerter-script\") pod \"iptables-alerter-r7mrf\" (UID: \"c85a95c6-4b03-47db-a7a3-f702a80fe01f\") " pod="openshift-network-operator/iptables-alerter-r7mrf"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597336    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-daemon-config\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597348    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-run-netns\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597389    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxckk\" (UniqueName: \"kubernetes.io/projected/0a0b6cf3-8303-4d98-b22f-f8a313721702-kube-api-access-fxckk\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597414    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-node-log\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597425    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb857965-3140-4b4e-982c-af2680bbeff3-env-overrides\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597430    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f82dd16e-0e8a-40ef-87bd-84560a6b9b71-serviceca\") pod \"node-ca-n8gtv\" (UID: \"f82dd16e-0e8a-40ef-87bd-84560a6b9b71\") " pod="openshift-image-registry/node-ca-n8gtv"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597437    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51727502-b532-46d0-828c-140b8eda2ba3-cnibin\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597472    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-device-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597505    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m5xt\" (UniqueName: \"kubernetes.io/projected/09961adc-4403-44ec-8c11-81938ae4fb16-kube-api-access-4m5xt\") pod \"periodic-gathering-m8znf-jgfmg\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") " pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597540    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-sysctl-d\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597586    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-sysctl-conf\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597621    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-run-ovn-kubernetes\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597648    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5ab344-e8d9-4b66-8e1f-6283db623632-config-volume\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:41.597684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597662    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/785f0704-ff0e-44e7-bc97-be771a84e03a-signing-cabundle\") pod \"service-ca-5fffc97bdf-4zrpt\" (UID: \"785f0704-ff0e-44e7-bc97-be771a84e03a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4zrpt"
Oct 30 20:19:41.598334 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597730    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-run\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.598334 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597754    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-sys\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.598334 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597780    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a0b6cf3-8303-4d98-b22f-f8a313721702-cni-binary-copy\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.598334 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597809    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-conf-dir\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.598334 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597864    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb857965-3140-4b4e-982c-af2680bbeff3-ovn-node-metrics-cert\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.598334 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597894    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl5jc\" (UniqueName: \"kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc\") pod \"network-check-target-d9xr4\" (UID: \"b9a64b7e-6436-4415-86f9-735350f24e35\") " pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:41.598334 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597934    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a0b6cf3-8303-4d98-b22f-f8a313721702-cni-binary-copy\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.598334 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.597950    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-tmp\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.598334 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.598035    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs\") pod \"network-metrics-daemon-vkj2j\" (UID: \"c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256\") " pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:41.598334 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.598040    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb857965-3140-4b4e-982c-af2680bbeff3-ovn-node-metrics-cert\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.598334 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.598069    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-tmp\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.598334 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.598106    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:41.598334 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.598129    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtjh9\" (UniqueName: \"kubernetes.io/projected/f82dd16e-0e8a-40ef-87bd-84560a6b9b71-kube-api-access-rtjh9\") pod \"node-ca-n8gtv\" (UID: \"f82dd16e-0e8a-40ef-87bd-84560a6b9b71\") " pod="openshift-image-registry/node-ca-n8gtv"
Oct 30 20:19:41.598334 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.598147    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-var-lib-cni-bin\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.598334 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.598188    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-kubelet\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.598334 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.598263    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:41.598334 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.598277    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-systemd\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.598758 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.598342    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-socket-dir-parent\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.598758 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.598364    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/09961adc-4403-44ec-8c11-81938ae4fb16-archives-path\") pod \"periodic-gathering-m8znf-jgfmg\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") " pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:19:41.598758 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.598439    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/09961adc-4403-44ec-8c11-81938ae4fb16-archives-path\") pod \"periodic-gathering-m8znf-jgfmg\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") " pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:19:41.612927 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.612905    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 20:19:41.613345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.613247    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 20:19:41.613345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.613281    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 20:19:41.613345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.613294    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 20:19:41.613345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.613310    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 20:19:41.613345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.613315    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jxfwk\""
Oct 30 20:19:41.613566 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.613315    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 20:19:41.613566 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.613390    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 20:19:41.613566 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.613529    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 20:19:41.613566 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.613539    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 20:19:41.613566 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.613550    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 20:19:41.613566 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.613531    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 20:19:41.619454 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.616230    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb857965-3140-4b4e-982c-af2680bbeff3-ovnkube-script-lib\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.619454 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.617588    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09961adc-4403-44ec-8c11-81938ae4fb16-service-ca-bundle\") pod \"periodic-gathering-m8znf-jgfmg\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") " pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:19:41.619454 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.618418    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q6sjr\""
Oct 30 20:19:41.619454 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.618629    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bp5wk\""
Oct 30 20:19:41.619454 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.618847    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 20:19:41.619454 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.619091    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vdqtm\""
Oct 30 20:19:41.619762 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.619537    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 20:19:41.619762 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.619598    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 20:19:41.619762 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.619617    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5ab344-e8d9-4b66-8e1f-6283db623632-config-volume\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:41.619913 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.619895    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 20:19:41.620267 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.620250    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/785f0704-ff0e-44e7-bc97-be771a84e03a-signing-cabundle\") pod \"service-ca-5fffc97bdf-4zrpt\" (UID: \"785f0704-ff0e-44e7-bc97-be771a84e03a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4zrpt"
Oct 30 20:19:41.626037 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.626023    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb8a13f8-5b1b-4a9d-965f-ad3e29051484-cert\") pod \"ingress-canary-mzdm7\" (UID: \"cb8a13f8-5b1b-4a9d-965f-ad3e29051484\") " pod="openshift-ingress-canary/ingress-canary-mzdm7"
Oct 30 20:19:41.626173 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.626157    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09961adc-4403-44ec-8c11-81938ae4fb16-trusted-ca-bundle\") pod \"periodic-gathering-m8znf-jgfmg\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") " pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:19:41.626896 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.626879    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/785f0704-ff0e-44e7-bc97-be771a84e03a-signing-key\") pod \"service-ca-5fffc97bdf-4zrpt\" (UID: \"785f0704-ff0e-44e7-bc97-be771a84e03a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4zrpt"
Oct 30 20:19:41.649620 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.649572    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6s8r\" (UniqueName: \"kubernetes.io/projected/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-kube-api-access-s6s8r\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.650098 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.650083    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqljx\" (UniqueName: \"kubernetes.io/projected/fb857965-3140-4b4e-982c-af2680bbeff3-kube-api-access-nqljx\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.650430 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.650414    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6tv5\" (UniqueName: \"kubernetes.io/projected/51727502-b532-46d0-828c-140b8eda2ba3-kube-api-access-s6tv5\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:41.650638 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.650622    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms86z\" (UniqueName: \"kubernetes.io/projected/98a309d0-a2a2-4ec4-82c4-5966823b6e06-kube-api-access-ms86z\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:41.652870 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.652852    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgw7z\" (UniqueName: \"kubernetes.io/projected/c85a95c6-4b03-47db-a7a3-f702a80fe01f-kube-api-access-tgw7z\") pod \"iptables-alerter-r7mrf\" (UID: \"c85a95c6-4b03-47db-a7a3-f702a80fe01f\") " pod="openshift-network-operator/iptables-alerter-r7mrf"
Oct 30 20:19:41.653408 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.653388    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq5ft\" (UniqueName: \"kubernetes.io/projected/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-kube-api-access-vq5ft\") pod \"network-metrics-daemon-vkj2j\" (UID: \"c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256\") " pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:41.653503 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.653488    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nfhx\" (UniqueName: \"kubernetes.io/projected/cb8a13f8-5b1b-4a9d-965f-ad3e29051484-kube-api-access-6nfhx\") pod \"ingress-canary-mzdm7\" (UID: \"cb8a13f8-5b1b-4a9d-965f-ad3e29051484\") " pod="openshift-ingress-canary/ingress-canary-mzdm7"
Oct 30 20:19:41.653575 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.653561    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c2c9\" (UniqueName: \"kubernetes.io/projected/bd71c9e1-948b-41cc-9d05-a65a556c8ea7-kube-api-access-4c2c9\") pod \"node-resolver-jbb5l\" (UID: \"bd71c9e1-948b-41cc-9d05-a65a556c8ea7\") " pod="openshift-dns/node-resolver-jbb5l"
Oct 30 20:19:41.654496 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.654481    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wgv9\" (UniqueName: \"kubernetes.io/projected/4d5ab344-e8d9-4b66-8e1f-6283db623632-kube-api-access-7wgv9\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:41.654640 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.654626    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl5jc\" (UniqueName: \"kubernetes.io/projected/b9a64b7e-6436-4415-86f9-735350f24e35-kube-api-access-vl5jc\") pod \"network-check-target-d9xr4\" (UID: \"b9a64b7e-6436-4415-86f9-735350f24e35\") " pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:41.654939 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.654925    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtjh9\" (UniqueName: \"kubernetes.io/projected/f82dd16e-0e8a-40ef-87bd-84560a6b9b71-kube-api-access-rtjh9\") pod \"node-ca-n8gtv\" (UID: \"f82dd16e-0e8a-40ef-87bd-84560a6b9b71\") " pod="openshift-image-registry/node-ca-n8gtv"
Oct 30 20:19:41.655518 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.655505    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxckk\" (UniqueName: \"kubernetes.io/projected/0a0b6cf3-8303-4d98-b22f-f8a313721702-kube-api-access-fxckk\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.655799 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.655779    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zkb7\" (UniqueName: \"kubernetes.io/projected/785f0704-ff0e-44e7-bc97-be771a84e03a-kube-api-access-4zkb7\") pod \"service-ca-5fffc97bdf-4zrpt\" (UID: \"785f0704-ff0e-44e7-bc97-be771a84e03a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4zrpt"
Oct 30 20:19:41.655923 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.655909    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kknl7\" (UniqueName: \"kubernetes.io/projected/3e789fe9-f262-4147-b4ee-ec42835f4ed0-kube-api-access-kknl7\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:41.655952 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.655917    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m5xt\" (UniqueName: \"kubernetes.io/projected/09961adc-4403-44ec-8c11-81938ae4fb16-kube-api-access-4m5xt\") pod \"periodic-gathering-m8znf-jgfmg\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") " pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:19:41.698882 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.698862    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-etc-openvswitch\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.698960 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.698886    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-cni-dir\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.698960 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.698901    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:41.698960 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.698915    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-socket-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:41.699093 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.698964    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:41.699093 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.698965    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-run-k8s-cni-cncf-io\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.699093 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.698973    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-etc-openvswitch\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.699093 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.698979    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-cni-dir\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.699093 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.698996    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-run-k8s-cni-cncf-io\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.699093 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699027    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-run-multus-certs\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.699093 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699024    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-socket-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:41.699093 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699052    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-etc-kubernetes\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.699093 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699060    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-run-multus-certs\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.699093 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699084    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c85a95c6-4b03-47db-a7a3-f702a80fe01f-host-slash\") pod \"iptables-alerter-r7mrf\" (UID: \"c85a95c6-4b03-47db-a7a3-f702a80fe01f\") " pod="openshift-network-operator/iptables-alerter-r7mrf"
Oct 30 20:19:41.699533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699113    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-etc-kubernetes\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.699533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699147    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-var-lib-openvswitch\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.699533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699190    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c85a95c6-4b03-47db-a7a3-f702a80fe01f-host-slash\") pod \"iptables-alerter-r7mrf\" (UID: \"c85a95c6-4b03-47db-a7a3-f702a80fe01f\") " pod="openshift-network-operator/iptables-alerter-r7mrf"
Oct 30 20:19:41.699533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699198    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-log-socket\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.699533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699234    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-var-lib-openvswitch\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.699533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699282    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-log-socket\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.699533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699278    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.699533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699340    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.699533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699393    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-etc-selinux\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:41.699533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699417    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-sys-fs\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:41.699533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699427    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-etc-selinux\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:41.699533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699441    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-sysconfig\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.699533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699460    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-sys-fs\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:41.699533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699464    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-registration-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:41.699533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699485    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-sysconfig\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.699533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699493    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-registration-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:41.699533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699487    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-lib-modules\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.699533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699517    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-var-lib-kubelet\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.699533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699540    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bd71c9e1-948b-41cc-9d05-a65a556c8ea7-hosts-file\") pod \"node-resolver-jbb5l\" (UID: \"bd71c9e1-948b-41cc-9d05-a65a556c8ea7\") " pod="openshift-dns/node-resolver-jbb5l"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699553    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-systemd-units\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699568    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-run-netns\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699582    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-node-log\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699583    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-lib-modules\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699619    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-var-lib-kubelet\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699627    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bd71c9e1-948b-41cc-9d05-a65a556c8ea7-hosts-file\") pod \"node-resolver-jbb5l\" (UID: \"bd71c9e1-948b-41cc-9d05-a65a556c8ea7\") " pod="openshift-dns/node-resolver-jbb5l"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699621    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-run-netns\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699635    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-node-log\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699641    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-systemd-units\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699656    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51727502-b532-46d0-828c-140b8eda2ba3-cnibin\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699675    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-device-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699690    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-sysctl-d\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699703    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-sysctl-conf\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699711    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51727502-b532-46d0-828c-140b8eda2ba3-cnibin\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699717    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3e789fe9-f262-4147-b4ee-ec42835f4ed0-device-dir\") pod \"aws-ebs-csi-driver-node-9pkpg\" (UID: \"3e789fe9-f262-4147-b4ee-ec42835f4ed0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9pkpg"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699725    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-run-ovn-kubernetes\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699746    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-run\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699752    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-run-ovn-kubernetes\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699764    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-sys\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699783    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-conf-dir\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699792    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-run\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699805    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs\") pod \"network-metrics-daemon-vkj2j\" (UID: \"c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256\") " pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699809    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-sysctl-d\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699828    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-var-lib-cni-bin\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699831    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-sys\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699832    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-sysctl-conf\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699846    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-conf-dir\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699849    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-kubelet\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699869    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-var-lib-cni-bin\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699877    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-systemd\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699877    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-kubelet\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699902    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-systemd\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699908    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-socket-dir-parent\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699936    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699958    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-system-cni-dir\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699969    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-multus-socket-dir-parent\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.699981    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-run-netns\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700004    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-system-cni-dir\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700004    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-run-openvswitch\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:41.700025    4471 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700030    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f399aaa3-b634-40e9-9ca7-0941f0ec2925-kubelet-config\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700060    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-cnibin\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700063    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-run-netns\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700065    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f399aaa3-b634-40e9-9ca7-0941f0ec2925-kubelet-config\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700034    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-run-openvswitch\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700083    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f399aaa3-b634-40e9-9ca7-0941f0ec2925-dbus\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:41.700097    4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls podName:98a309d0-a2a2-4ec4-82c4-5966823b6e06 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:42.200077845 +0000 UTC m=+1.878706447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6pstp" (UID: "98a309d0-a2a2-4ec4-82c4-5966823b6e06") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700118    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f399aaa3-b634-40e9-9ca7-0941f0ec2925-dbus\") pod \"global-pull-secret-syncer-jkz4p\" (UID: \"f399aaa3-b634-40e9-9ca7-0941f0ec2925\") " pod="kube-system/global-pull-secret-syncer-jkz4p"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700126    4471 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700138    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-cnibin\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700139    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-hostroot\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700169    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-hostroot\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700169    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-run-ovn\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700195    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-run-ovn\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700206    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-cni-bin\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700232    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-cni-netd\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700246    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-cni-bin\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.700212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700255    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700266    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-cni-netd\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700278    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-var-lib-cni-multus\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700298    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-var-lib-kubelet\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700314    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-run-systemd\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:41.700339    4471 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700355    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-kubernetes\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700361    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-var-lib-kubelet\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700363    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a0b6cf3-8303-4d98-b22f-f8a313721702-host-var-lib-cni-multus\") pod \"multus-psqxp\" (UID: \"0a0b6cf3-8303-4d98-b22f-f8a313721702\") " pod="openshift-multus/multus-psqxp"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700378    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-host\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:41.700392    4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls podName:4d5ab344-e8d9-4b66-8e1f-6283db623632 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:42.200376497 +0000 UTC m=+1.879005101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls") pod "dns-default-jlxjn" (UID: "4d5ab344-e8d9-4b66-8e1f-6283db623632") : secret "dns-default-metrics-tls" not found
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700389    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-run-systemd\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700421    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-host\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700429    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98a309d0-a2a2-4ec4-82c4-5966823b6e06-crio-socket\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700440    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-kubernetes\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700453    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-slash\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700458    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98a309d0-a2a2-4ec4-82c4-5966823b6e06-crio-socket\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700472    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51727502-b532-46d0-828c-140b8eda2ba3-system-cni-dir\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700487    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51727502-b532-46d0-828c-140b8eda2ba3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700492    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb857965-3140-4b4e-982c-af2680bbeff3-host-slash\") pod \"ovnkube-node-frx59\" (UID: \"fb857965-3140-4b4e-982c-af2680bbeff3\") " pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700501    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f82dd16e-0e8a-40ef-87bd-84560a6b9b71-host\") pod \"node-ca-n8gtv\" (UID: \"f82dd16e-0e8a-40ef-87bd-84560a6b9b71\") " pod="openshift-image-registry/node-ca-n8gtv"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700518    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51727502-b532-46d0-828c-140b8eda2ba3-system-cni-dir\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700521    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-modprobe-d\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700548    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f82dd16e-0e8a-40ef-87bd-84560a6b9b71-host\") pod \"node-ca-n8gtv\" (UID: \"f82dd16e-0e8a-40ef-87bd-84560a6b9b71\") " pod="openshift-image-registry/node-ca-n8gtv"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700601    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f9aa5d5-0be0-4914-a6cc-bd54151feb49-etc-modprobe-d\") pod \"tuned-cwm29\" (UID: \"5f9aa5d5-0be0-4914-a6cc-bd54151feb49\") " pod="openshift-cluster-node-tuning-operator/tuned-cwm29"
Oct 30 20:19:41.701947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.700581    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51727502-b532-46d0-828c-140b8eda2ba3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2tgdb\" (UID: \"51727502-b532-46d0-828c-140b8eda2ba3\") " pod="openshift-multus/multus-additional-cni-plugins-2tgdb"
Oct 30 20:19:41.702632 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.702178    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256-metrics-certs\") pod \"network-metrics-daemon-vkj2j\" (UID: \"c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256\") " pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:41.817117 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.817093    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vkj2j"
Oct 30 20:19:41.962195 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:41.962128    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mzdm7_cb8a13f8-5b1b-4a9d-965f-ad3e29051484/serve-healthcheck-canary/0.log"
Oct 30 20:19:42.186089 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:42.186064    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vkj2j"]
Oct 30 20:19:42.190306 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:42.190269    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc36aa5d4_3db7_4d0d_b53f_fa4bdcde7256.slice/crio-35f111d6dd74a40274edd9a20707e24caa27740ded37a49d1af7930204951801 WatchSource:0}: Error finding container 35f111d6dd74a40274edd9a20707e24caa27740ded37a49d1af7930204951801: Status 404 returned error can't find the container with id 35f111d6dd74a40274edd9a20707e24caa27740ded37a49d1af7930204951801
Oct 30 20:19:42.192396 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:42.192380    4471 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:19:42.205475 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:42.205450    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:42.205541 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:42.205484    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:42.205598 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:42.205584    4471 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:42.205647 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:42.205639    4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls podName:98a309d0-a2a2-4ec4-82c4-5966823b6e06 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:43.205626716 +0000 UTC m=+2.884255322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6pstp" (UID: "98a309d0-a2a2-4ec4-82c4-5966823b6e06") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:42.205682 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:42.205584    4471 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:19:42.205682 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:42.205680    4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls podName:4d5ab344-e8d9-4b66-8e1f-6283db623632 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:43.20567331 +0000 UTC m=+2.884301904 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls") pod "dns-default-jlxjn" (UID: "4d5ab344-e8d9-4b66-8e1f-6283db623632") : secret "dns-default-metrics-tls" not found
Oct 30 20:19:42.463083 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:42.463053    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vkj2j" event={"ID":"c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256","Type":"ContainerStarted","Data":"35f111d6dd74a40274edd9a20707e24caa27740ded37a49d1af7930204951801"}
Oct 30 20:19:43.212802 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:43.212767    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:43.212979 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:43.212823    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:43.212979 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:43.212937    4471 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:19:43.212979 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:43.212956    4471 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:43.213117 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:43.213006    4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls podName:4d5ab344-e8d9-4b66-8e1f-6283db623632 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:45.212994379 +0000 UTC m=+4.891622985 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls") pod "dns-default-jlxjn" (UID: "4d5ab344-e8d9-4b66-8e1f-6283db623632") : secret "dns-default-metrics-tls" not found
Oct 30 20:19:43.213117 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:43.213035    4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls podName:98a309d0-a2a2-4ec4-82c4-5966823b6e06 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:45.213017396 +0000 UTC m=+4.891645995 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6pstp" (UID: "98a309d0-a2a2-4ec4-82c4-5966823b6e06") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:44.140750 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:44.140718    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:44.323074 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:44.323044    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:44.470562 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:44.470485    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vkj2j" event={"ID":"c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256","Type":"ContainerStarted","Data":"c5bc0109045af17680334eca5d799f527378fb43bb442c76c795ad47b8c49876"}
Oct 30 20:19:44.470562 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:44.470544    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vkj2j" event={"ID":"c36aa5d4-3db7-4d0d-b53f-fa4bdcde7256","Type":"ContainerStarted","Data":"77d25f50b69c1c824cfe4e26b98b2cf3e03db3077370eb1e8d869bea6c6699ad"}
Oct 30 20:19:44.470770 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:44.470571    4471 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:19:45.226310 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:45.226270    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:45.226687 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:45.226313    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:45.226687 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:45.226417    4471 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:45.226687 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:45.226446    4471 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 20:19:45.226687 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:45.226488    4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls podName:98a309d0-a2a2-4ec4-82c4-5966823b6e06 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:49.226472592 +0000 UTC m=+8.905101185 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6pstp" (UID: "98a309d0-a2a2-4ec4-82c4-5966823b6e06") : secret "insights-runtime-extractor-tls" not found
Oct 30 20:19:45.226687 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:19:45.226504    4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls podName:4d5ab344-e8d9-4b66-8e1f-6283db623632 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:49.226497582 +0000 UTC m=+8.905126175 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls") pod "dns-default-jlxjn" (UID: "4d5ab344-e8d9-4b66-8e1f-6283db623632") : secret "dns-default-metrics-tls" not found
Oct 30 20:19:46.116525 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:46.116490    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:19:47.165899 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:47.165878    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-frx59"
Oct 30 20:19:49.254654 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:49.254617    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:49.255122 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:49.254676    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:49.256969 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:49.256943    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d5ab344-e8d9-4b66-8e1f-6283db623632-metrics-tls\") pod \"dns-default-jlxjn\" (UID: \"4d5ab344-e8d9-4b66-8e1f-6283db623632\") " pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:49.257073 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:49.257055    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98a309d0-a2a2-4ec4-82c4-5966823b6e06-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6pstp\" (UID: \"98a309d0-a2a2-4ec4-82c4-5966823b6e06\") " pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:49.319039 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:49.319014    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6pstp"
Oct 30 20:19:49.319137 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:49.319087    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:49.825902 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:49.825873    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jlxjn"]
Oct 30 20:19:49.828745 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:49.828721    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a309d0_a2a2_4ec4_82c4_5966823b6e06.slice/crio-88d8d6874d8aa43cd213df5a868ce32c585c03140699d199c22b2a1d5399ccd9 WatchSource:0}: Error finding container 88d8d6874d8aa43cd213df5a868ce32c585c03140699d199c22b2a1d5399ccd9: Status 404 returned error can't find the container with id 88d8d6874d8aa43cd213df5a868ce32c585c03140699d199c22b2a1d5399ccd9
Oct 30 20:19:49.829287 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:49.829269    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d5ab344_e8d9_4b66_8e1f_6283db623632.slice/crio-b5ef3ed8efd24fc0622f0247e6ae6bcf787d5e800ffbeb793ebca52df63044d9 WatchSource:0}: Error finding container b5ef3ed8efd24fc0622f0247e6ae6bcf787d5e800ffbeb793ebca52df63044d9: Status 404 returned error can't find the container with id b5ef3ed8efd24fc0622f0247e6ae6bcf787d5e800ffbeb793ebca52df63044d9
Oct 30 20:19:49.875189 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:49.875167    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6pstp"]
Oct 30 20:19:50.489301 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:50.489266    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jlxjn" event={"ID":"4d5ab344-e8d9-4b66-8e1f-6283db623632","Type":"ContainerStarted","Data":"b5ef3ed8efd24fc0622f0247e6ae6bcf787d5e800ffbeb793ebca52df63044d9"}
Oct 30 20:19:50.490754 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:50.490718    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6pstp" event={"ID":"98a309d0-a2a2-4ec4-82c4-5966823b6e06","Type":"ContainerStarted","Data":"01930a4f5ad55270b34881ff55f2080f5ca4e8ac0b8f757c5879339531b2ef4f"}
Oct 30 20:19:50.490877 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:50.490773    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6pstp" event={"ID":"98a309d0-a2a2-4ec4-82c4-5966823b6e06","Type":"ContainerStarted","Data":"88d8d6874d8aa43cd213df5a868ce32c585c03140699d199c22b2a1d5399ccd9"}
Oct 30 20:19:50.968147 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:50.968116    4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-34.us-west-1.compute.internal" event="NodeReady"
Oct 30 20:19:51.497732 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:51.497698    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qxwhp"
Oct 30 20:19:52.501396 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.501369    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-28mfw"]
Oct 30 20:19:52.524145 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.524108    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jlxjn" event={"ID":"4d5ab344-e8d9-4b66-8e1f-6283db623632","Type":"ContainerStarted","Data":"01ed72475c9848208bdd91e3502d3b82d8991b118db2ef4dde232c5e2bdc6fa5"}
Oct 30 20:19:52.524260 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.524148    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6pstp" event={"ID":"98a309d0-a2a2-4ec4-82c4-5966823b6e06","Type":"ContainerStarted","Data":"ea84d074f6a4ba947a190774a640d0313d8fdc4e673cbd82fd5be874d0f7c8c4"}
Oct 30 20:19:52.524302 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.524265    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-28mfw"
Oct 30 20:19:52.572165 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.572139    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-849b5846cf-mn7w4"]
Oct 30 20:19:52.576682 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.576509    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8a819e6-7a94-4fa3-9035-7796e390f228-tmp\") pod \"downloads-6dd789c8b7-28mfw\" (UID: \"b8a819e6-7a94-4fa3-9035-7796e390f228\") " pod="openshift-console/downloads-6dd789c8b7-28mfw"
Oct 30 20:19:52.576682 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.576598    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pztl8\" (UniqueName: \"kubernetes.io/projected/b8a819e6-7a94-4fa3-9035-7796e390f228-kube-api-access-pztl8\") pod \"downloads-6dd789c8b7-28mfw\" (UID: \"b8a819e6-7a94-4fa3-9035-7796e390f228\") " pod="openshift-console/downloads-6dd789c8b7-28mfw"
Oct 30 20:19:52.591260 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.591241    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.599170 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.599150    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 20:19:52.599284 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.599227    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 20:19:52.620282 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.620026    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-q929l\""
Oct 30 20:19:52.621530 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.621510    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-28mfw"]
Oct 30 20:19:52.657137 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.657073    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-849b5846cf-mn7w4"]
Oct 30 20:19:52.677284 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.677256    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/31974b7a-cd1b-48ec-a8bf-61025daffada-trusted-ca\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.677426 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.677304    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/31974b7a-cd1b-48ec-a8bf-61025daffada-installation-pull-secrets\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.677426 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.677416    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nl4f\" (UniqueName: \"kubernetes.io/projected/31974b7a-cd1b-48ec-a8bf-61025daffada-kube-api-access-6nl4f\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.677542 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.677454    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8a819e6-7a94-4fa3-9035-7796e390f228-tmp\") pod \"downloads-6dd789c8b7-28mfw\" (UID: \"b8a819e6-7a94-4fa3-9035-7796e390f228\") " pod="openshift-console/downloads-6dd789c8b7-28mfw"
Oct 30 20:19:52.677542 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.677485    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/31974b7a-cd1b-48ec-a8bf-61025daffada-ca-trust-extracted\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.677672 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.677649    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/31974b7a-cd1b-48ec-a8bf-61025daffada-bound-sa-token\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.677722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.677690    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31974b7a-cd1b-48ec-a8bf-61025daffada-registry-tls\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.677722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.677716    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/31974b7a-cd1b-48ec-a8bf-61025daffada-registry-certificates\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.677820 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.677782    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pztl8\" (UniqueName: \"kubernetes.io/projected/b8a819e6-7a94-4fa3-9035-7796e390f228-kube-api-access-pztl8\") pod \"downloads-6dd789c8b7-28mfw\" (UID: \"b8a819e6-7a94-4fa3-9035-7796e390f228\") " pod="openshift-console/downloads-6dd789c8b7-28mfw"
Oct 30 20:19:52.677874 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.677828    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/31974b7a-cd1b-48ec-a8bf-61025daffada-image-registry-private-configuration\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.677874 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.677832    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8a819e6-7a94-4fa3-9035-7796e390f228-tmp\") pod \"downloads-6dd789c8b7-28mfw\" (UID: \"b8a819e6-7a94-4fa3-9035-7796e390f228\") " pod="openshift-console/downloads-6dd789c8b7-28mfw"
Oct 30 20:19:52.720014 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.719966    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 20:19:52.720221 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.720200    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 20:19:52.727775 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.727750    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 20:19:52.727875 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.727790    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hbnvb\""
Oct 30 20:19:52.727875 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.727806    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 20:19:52.757707 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.757679    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pztl8\" (UniqueName: \"kubernetes.io/projected/b8a819e6-7a94-4fa3-9035-7796e390f228-kube-api-access-pztl8\") pod \"downloads-6dd789c8b7-28mfw\" (UID: \"b8a819e6-7a94-4fa3-9035-7796e390f228\") " pod="openshift-console/downloads-6dd789c8b7-28mfw"
Oct 30 20:19:52.778198 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.778175    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/31974b7a-cd1b-48ec-a8bf-61025daffada-bound-sa-token\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.778292 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.778205    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31974b7a-cd1b-48ec-a8bf-61025daffada-registry-tls\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.778292 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.778221    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/31974b7a-cd1b-48ec-a8bf-61025daffada-registry-certificates\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.778394 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.778372    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/31974b7a-cd1b-48ec-a8bf-61025daffada-image-registry-private-configuration\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.778427 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.778414    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/31974b7a-cd1b-48ec-a8bf-61025daffada-trusted-ca\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.778553 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.778533    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/31974b7a-cd1b-48ec-a8bf-61025daffada-installation-pull-secrets\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.778595 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.778571    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nl4f\" (UniqueName: \"kubernetes.io/projected/31974b7a-cd1b-48ec-a8bf-61025daffada-kube-api-access-6nl4f\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.778648 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.778613    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/31974b7a-cd1b-48ec-a8bf-61025daffada-ca-trust-extracted\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.779021 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.778999    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/31974b7a-cd1b-48ec-a8bf-61025daffada-ca-trust-extracted\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.779271 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.779024    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/31974b7a-cd1b-48ec-a8bf-61025daffada-registry-certificates\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.779603 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.779580    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/31974b7a-cd1b-48ec-a8bf-61025daffada-trusted-ca\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.780740 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.780720    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31974b7a-cd1b-48ec-a8bf-61025daffada-registry-tls\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.780852 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.780802    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/31974b7a-cd1b-48ec-a8bf-61025daffada-image-registry-private-configuration\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.780907 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.780876    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/31974b7a-cd1b-48ec-a8bf-61025daffada-installation-pull-secrets\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.838696 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.838674    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/31974b7a-cd1b-48ec-a8bf-61025daffada-bound-sa-token\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.838904 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.838886    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nl4f\" (UniqueName: \"kubernetes.io/projected/31974b7a-cd1b-48ec-a8bf-61025daffada-kube-api-access-6nl4f\") pod \"image-registry-849b5846cf-mn7w4\" (UID: \"31974b7a-cd1b-48ec-a8bf-61025daffada\") " pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:52.868268 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.868248    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-28mfw"
Oct 30 20:19:52.908525 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:52.908479    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:53.314790 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:53.314705    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-28mfw"]
Oct 30 20:19:53.317978 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:53.317954    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-849b5846cf-mn7w4"]
Oct 30 20:19:53.318633 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:53.318596    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a819e6_7a94_4fa3_9035_7796e390f228.slice/crio-c1f0620b71f7a59a94b05752e8af00ab96554833ff7d23331080fa6feb84b3ee WatchSource:0}: Error finding container c1f0620b71f7a59a94b05752e8af00ab96554833ff7d23331080fa6feb84b3ee: Status 404 returned error can't find the container with id c1f0620b71f7a59a94b05752e8af00ab96554833ff7d23331080fa6feb84b3ee
Oct 30 20:19:53.322563 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:19:53.322544    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31974b7a_cd1b_48ec_a8bf_61025daffada.slice/crio-0fbd932a7ffd1f3db09dde2303bfbd9e561bd1d091b4ec6fd3838f28067949bc WatchSource:0}: Error finding container 0fbd932a7ffd1f3db09dde2303bfbd9e561bd1d091b4ec6fd3838f28067949bc: Status 404 returned error can't find the container with id 0fbd932a7ffd1f3db09dde2303bfbd9e561bd1d091b4ec6fd3838f28067949bc
Oct 30 20:19:53.508798 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:53.508758    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-28mfw" event={"ID":"b8a819e6-7a94-4fa3-9035-7796e390f228","Type":"ContainerStarted","Data":"c1f0620b71f7a59a94b05752e8af00ab96554833ff7d23331080fa6feb84b3ee"}
Oct 30 20:19:53.512184 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:53.512151    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-849b5846cf-mn7w4" event={"ID":"31974b7a-cd1b-48ec-a8bf-61025daffada","Type":"ContainerStarted","Data":"aa1263a1bb550aa2a4be1678ebf1b80cec6b418d8895cb480dca78b6ae2d5c9c"}
Oct 30 20:19:53.512356 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:53.512206    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-849b5846cf-mn7w4" event={"ID":"31974b7a-cd1b-48ec-a8bf-61025daffada","Type":"ContainerStarted","Data":"0fbd932a7ffd1f3db09dde2303bfbd9e561bd1d091b4ec6fd3838f28067949bc"}
Oct 30 20:19:53.512436 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:53.512373    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:19:53.514121 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:53.514100    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jlxjn" event={"ID":"4d5ab344-e8d9-4b66-8e1f-6283db623632","Type":"ContainerStarted","Data":"235fc4c34b8816ba8ea02ba658261bf05c0b0734802b72ba72b1a01009930060"}
Oct 30 20:19:53.514251 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:53.514238    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:19:53.644083 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:53.643971    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-849b5846cf-mn7w4" podStartSLOduration=1.643952488 podStartE2EDuration="1.643952488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:53.643253823 +0000 UTC m=+13.321882459" watchObservedRunningTime="2026-10-30 20:19:53.643952488 +0000 UTC m=+13.322581104"
Oct 30 20:19:54.519272 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:19:54.519192    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6pstp" event={"ID":"98a309d0-a2a2-4ec4-82c4-5966823b6e06","Type":"ContainerStarted","Data":"9f8c6b7845b38b3aaf5ec4b9e9adf222afebacc76aeb5c03a34bdec6e88abe77"}
Oct 30 20:20:00.955401 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:20:00.955366    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:20:03.522125 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:03.522092    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jlxjn"
Oct 30 20:20:07.091198 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.091160    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-68flc"]
Oct 30 20:20:07.093715 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.093687    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.179267 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.179234    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/28cd454f-5dd2-4451-b955-f09a0fadd91b-node-exporter-wtmp\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.179267 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.179265    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28cd454f-5dd2-4451-b955-f09a0fadd91b-metrics-client-ca\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.179524 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.179281    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb7vm\" (UniqueName: \"kubernetes.io/projected/28cd454f-5dd2-4451-b955-f09a0fadd91b-kube-api-access-fb7vm\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.179524 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.179346    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28cd454f-5dd2-4451-b955-f09a0fadd91b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.179524 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.179451    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/28cd454f-5dd2-4451-b955-f09a0fadd91b-root\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.179680 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.179534    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/28cd454f-5dd2-4451-b955-f09a0fadd91b-node-exporter-accelerators-collector-config\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.179680 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.179575    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/28cd454f-5dd2-4451-b955-f09a0fadd91b-node-exporter-textfile\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.179680 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.179608    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28cd454f-5dd2-4451-b955-f09a0fadd91b-sys\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.179680 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.179633    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/28cd454f-5dd2-4451-b955-f09a0fadd91b-node-exporter-tls\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.267185 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.267144    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 20:20:07.267185 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.267171    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 20:20:07.267441 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.267204    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n9jp6\""
Oct 30 20:20:07.267441 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.267347    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 20:20:07.267441 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.267382    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 20:20:07.267574 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.267455    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 20:20:07.267574 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.267461    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 20:20:07.282054 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.280615    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/28cd454f-5dd2-4451-b955-f09a0fadd91b-node-exporter-wtmp\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.282054 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.280649    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28cd454f-5dd2-4451-b955-f09a0fadd91b-metrics-client-ca\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.282054 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.280665    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb7vm\" (UniqueName: \"kubernetes.io/projected/28cd454f-5dd2-4451-b955-f09a0fadd91b-kube-api-access-fb7vm\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.282054 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.280688    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28cd454f-5dd2-4451-b955-f09a0fadd91b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.282054 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.280710    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/28cd454f-5dd2-4451-b955-f09a0fadd91b-root\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.282054 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.280745    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/28cd454f-5dd2-4451-b955-f09a0fadd91b-node-exporter-accelerators-collector-config\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.282054 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.280782    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/28cd454f-5dd2-4451-b955-f09a0fadd91b-node-exporter-textfile\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.282054 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.280813    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28cd454f-5dd2-4451-b955-f09a0fadd91b-sys\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.282054 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.280815    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/28cd454f-5dd2-4451-b955-f09a0fadd91b-node-exporter-wtmp\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.282054 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.280835    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/28cd454f-5dd2-4451-b955-f09a0fadd91b-node-exporter-tls\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.282054 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.281081    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/28cd454f-5dd2-4451-b955-f09a0fadd91b-root\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.282054 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.281233    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28cd454f-5dd2-4451-b955-f09a0fadd91b-sys\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.282054 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.281382    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/28cd454f-5dd2-4451-b955-f09a0fadd91b-node-exporter-textfile\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.282054 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.281388    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28cd454f-5dd2-4451-b955-f09a0fadd91b-metrics-client-ca\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.282054 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.281388    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/28cd454f-5dd2-4451-b955-f09a0fadd91b-node-exporter-accelerators-collector-config\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.283978 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.283953    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/28cd454f-5dd2-4451-b955-f09a0fadd91b-node-exporter-tls\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.284089 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.283957    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28cd454f-5dd2-4451-b955-f09a0fadd91b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.337094 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.337064    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb7vm\" (UniqueName: \"kubernetes.io/projected/28cd454f-5dd2-4451-b955-f09a0fadd91b-kube-api-access-fb7vm\") pod \"node-exporter-68flc\" (UID: \"28cd454f-5dd2-4451-b955-f09a0fadd91b\") " pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.410524 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.410484    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-68flc"
Oct 30 20:20:07.420521 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:20:07.420490    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28cd454f_5dd2_4451_b955_f09a0fadd91b.slice/crio-7b6e8af12fcbb39eec99a1a6e9d0d8359065e0a59756087306b2bbe030d4787b WatchSource:0}: Error finding container 7b6e8af12fcbb39eec99a1a6e9d0d8359065e0a59756087306b2bbe030d4787b: Status 404 returned error can't find the container with id 7b6e8af12fcbb39eec99a1a6e9d0d8359065e0a59756087306b2bbe030d4787b
Oct 30 20:20:07.564230 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:07.564193    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-68flc" event={"ID":"28cd454f-5dd2-4451-b955-f09a0fadd91b","Type":"ContainerStarted","Data":"7b6e8af12fcbb39eec99a1a6e9d0d8359065e0a59756087306b2bbe030d4787b"}
Oct 30 20:20:09.574162 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:09.574121    4471 generic.go:356] "Generic (PLEG): container finished" podID="28cd454f-5dd2-4451-b955-f09a0fadd91b" containerID="f1646eea187134db148b980cb1d9cc708b81a4301830cb5178ecd1854af67845" exitCode=0
Oct 30 20:20:09.574654 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:09.574186    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-68flc" event={"ID":"28cd454f-5dd2-4451-b955-f09a0fadd91b","Type":"ContainerDied","Data":"f1646eea187134db148b980cb1d9cc708b81a4301830cb5178ecd1854af67845"}
Oct 30 20:20:10.579991 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:10.579955    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-68flc" event={"ID":"28cd454f-5dd2-4451-b955-f09a0fadd91b","Type":"ContainerStarted","Data":"251851e782c2272b3834cadb30c618d72632d1e3c12eec58f1d7c1fcd7d65109"}
Oct 30 20:20:10.580424 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:10.580015    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-68flc" event={"ID":"28cd454f-5dd2-4451-b955-f09a0fadd91b","Type":"ContainerStarted","Data":"eca2a944070cc3b8970ac80700e11141c869ec95286f0eda0bbcca38505fc063"}
Oct 30 20:20:10.746725 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:10.746664    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-68flc" podStartSLOduration=2.266575638 podStartE2EDuration="3.746646326s" totalImagesPullingTime="1.399142784s" totalInitContainerRuntime="80.927904ms" isStatefulPod=true podCreationTimestamp="2026-10-30 20:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:10.746062973 +0000 UTC m=+30.424691589" watchObservedRunningTime="2026-10-30 20:20:10.746646326 +0000 UTC m=+30.425274940"
Oct 30 20:20:11.880433 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:11.880398    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6d486dff54-jwr4n"]
Oct 30 20:20:11.882450 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:11.882422    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:11.936810 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:11.936784    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 20:20:11.940481 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:11.937176    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 20:20:11.940481 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:11.939737    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 20:20:11.940481 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:11.939826    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 20:20:11.940481 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:11.939937    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-pbnk9\""
Oct 30 20:20:11.940481 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:11.940066    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-14v5bjm37r0em\""
Oct 30 20:20:11.945409 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:11.945385    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6d486dff54-jwr4n"]
Oct 30 20:20:12.021178 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.021146    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2qkn\" (UniqueName: \"kubernetes.io/projected/7aee5968-6fa2-4193-858e-3d355c07f4b7-kube-api-access-p2qkn\") pod \"metrics-server-6d486dff54-jwr4n\" (UID: \"7aee5968-6fa2-4193-858e-3d355c07f4b7\") " pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:12.021360 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.021202    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7aee5968-6fa2-4193-858e-3d355c07f4b7-audit-log\") pod \"metrics-server-6d486dff54-jwr4n\" (UID: \"7aee5968-6fa2-4193-858e-3d355c07f4b7\") " pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:12.021360 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.021298    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7aee5968-6fa2-4193-858e-3d355c07f4b7-client-ca-bundle\") pod \"metrics-server-6d486dff54-jwr4n\" (UID: \"7aee5968-6fa2-4193-858e-3d355c07f4b7\") " pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:12.021470 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.021392    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7aee5968-6fa2-4193-858e-3d355c07f4b7-metrics-server-audit-profiles\") pod \"metrics-server-6d486dff54-jwr4n\" (UID: \"7aee5968-6fa2-4193-858e-3d355c07f4b7\") " pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:12.021470 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.021451    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aee5968-6fa2-4193-858e-3d355c07f4b7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d486dff54-jwr4n\" (UID: \"7aee5968-6fa2-4193-858e-3d355c07f4b7\") " pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:12.021533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.021486    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7aee5968-6fa2-4193-858e-3d355c07f4b7-secret-metrics-server-client-certs\") pod \"metrics-server-6d486dff54-jwr4n\" (UID: \"7aee5968-6fa2-4193-858e-3d355c07f4b7\") " pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:12.021533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.021512    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7aee5968-6fa2-4193-858e-3d355c07f4b7-secret-metrics-server-tls\") pod \"metrics-server-6d486dff54-jwr4n\" (UID: \"7aee5968-6fa2-4193-858e-3d355c07f4b7\") " pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:12.122648 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.122624    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aee5968-6fa2-4193-858e-3d355c07f4b7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d486dff54-jwr4n\" (UID: \"7aee5968-6fa2-4193-858e-3d355c07f4b7\") " pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:12.122726 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.122670    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7aee5968-6fa2-4193-858e-3d355c07f4b7-secret-metrics-server-client-certs\") pod \"metrics-server-6d486dff54-jwr4n\" (UID: \"7aee5968-6fa2-4193-858e-3d355c07f4b7\") " pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:12.122861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.122837    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7aee5968-6fa2-4193-858e-3d355c07f4b7-secret-metrics-server-tls\") pod \"metrics-server-6d486dff54-jwr4n\" (UID: \"7aee5968-6fa2-4193-858e-3d355c07f4b7\") " pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:12.122927 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.122913    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2qkn\" (UniqueName: \"kubernetes.io/projected/7aee5968-6fa2-4193-858e-3d355c07f4b7-kube-api-access-p2qkn\") pod \"metrics-server-6d486dff54-jwr4n\" (UID: \"7aee5968-6fa2-4193-858e-3d355c07f4b7\") " pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:12.123044 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.122965    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7aee5968-6fa2-4193-858e-3d355c07f4b7-audit-log\") pod \"metrics-server-6d486dff54-jwr4n\" (UID: \"7aee5968-6fa2-4193-858e-3d355c07f4b7\") " pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:12.123044 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.122990    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7aee5968-6fa2-4193-858e-3d355c07f4b7-client-ca-bundle\") pod \"metrics-server-6d486dff54-jwr4n\" (UID: \"7aee5968-6fa2-4193-858e-3d355c07f4b7\") " pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:12.123044 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.123020    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7aee5968-6fa2-4193-858e-3d355c07f4b7-metrics-server-audit-profiles\") pod \"metrics-server-6d486dff54-jwr4n\" (UID: \"7aee5968-6fa2-4193-858e-3d355c07f4b7\") " pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:12.123435 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.123409    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aee5968-6fa2-4193-858e-3d355c07f4b7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d486dff54-jwr4n\" (UID: \"7aee5968-6fa2-4193-858e-3d355c07f4b7\") " pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:12.123596 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.123537    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7aee5968-6fa2-4193-858e-3d355c07f4b7-audit-log\") pod \"metrics-server-6d486dff54-jwr4n\" (UID: \"7aee5968-6fa2-4193-858e-3d355c07f4b7\") " pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:12.124217 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.124194    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7aee5968-6fa2-4193-858e-3d355c07f4b7-metrics-server-audit-profiles\") pod \"metrics-server-6d486dff54-jwr4n\" (UID: \"7aee5968-6fa2-4193-858e-3d355c07f4b7\") " pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:12.125500 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.125478    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7aee5968-6fa2-4193-858e-3d355c07f4b7-secret-metrics-server-tls\") pod \"metrics-server-6d486dff54-jwr4n\" (UID: \"7aee5968-6fa2-4193-858e-3d355c07f4b7\") " pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:12.125703 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.125666    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7aee5968-6fa2-4193-858e-3d355c07f4b7-secret-metrics-server-client-certs\") pod \"metrics-server-6d486dff54-jwr4n\" (UID: \"7aee5968-6fa2-4193-858e-3d355c07f4b7\") " pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:12.125768 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.125758    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7aee5968-6fa2-4193-858e-3d355c07f4b7-client-ca-bundle\") pod \"metrics-server-6d486dff54-jwr4n\" (UID: \"7aee5968-6fa2-4193-858e-3d355c07f4b7\") " pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:12.178730 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.178702    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2qkn\" (UniqueName: \"kubernetes.io/projected/7aee5968-6fa2-4193-858e-3d355c07f4b7-kube-api-access-p2qkn\") pod \"metrics-server-6d486dff54-jwr4n\" (UID: \"7aee5968-6fa2-4193-858e-3d355c07f4b7\") " pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:12.193587 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.193567    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:12.590057 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.589973    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-28mfw" event={"ID":"b8a819e6-7a94-4fa3-9035-7796e390f228","Type":"ContainerStarted","Data":"5a9155f263ab8b854df589a5fae6f6e6a05f89faa4b7ee1aa74835e7b2629f77"}
Oct 30 20:20:12.590265 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.590238    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-28mfw"
Oct 30 20:20:12.607666 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.607642    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-28mfw"
Oct 30 20:20:12.611513 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.611486    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6d486dff54-jwr4n"]
Oct 30 20:20:12.615527 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:20:12.615498    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aee5968_6fa2_4193_858e_3d355c07f4b7.slice/crio-1928e04b3678f52b1ed709a0df25e0b24881b09787b99d84a496b6ec8b0c9de5 WatchSource:0}: Error finding container 1928e04b3678f52b1ed709a0df25e0b24881b09787b99d84a496b6ec8b0c9de5: Status 404 returned error can't find the container with id 1928e04b3678f52b1ed709a0df25e0b24881b09787b99d84a496b6ec8b0c9de5
Oct 30 20:20:12.707008 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:12.706955    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-28mfw" podStartSLOduration=1.917918612 podStartE2EDuration="20.706939258s" totalImagesPullingTime="18.789020646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:12.706541147 +0000 UTC m=+32.385169763" watchObservedRunningTime="2026-10-30 20:20:12.706939258 +0000 UTC m=+32.385567873"
Oct 30 20:20:13.595903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:13.595863    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n" event={"ID":"7aee5968-6fa2-4193-858e-3d355c07f4b7","Type":"ContainerStarted","Data":"1928e04b3678f52b1ed709a0df25e0b24881b09787b99d84a496b6ec8b0c9de5"}
Oct 30 20:20:14.525131 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:14.524524    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-849b5846cf-mn7w4"
Oct 30 20:20:15.607288 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:15.607252    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n" event={"ID":"7aee5968-6fa2-4193-858e-3d355c07f4b7","Type":"ContainerStarted","Data":"cbb90a0bbdfe7eafd6c9eaf0cde5b548c2aa879ec499c2c5c99fd4151ec36048"}
Oct 30 20:20:15.730174 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:15.730105    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n" podStartSLOduration=2.234032366 podStartE2EDuration="4.730084145s" totalImagesPullingTime="2.496051779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:15.728875658 +0000 UTC m=+35.407504275" watchObservedRunningTime="2026-10-30 20:20:15.730084145 +0000 UTC m=+35.408712760"
Oct 30 20:20:16.120234 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:16.120201    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-d9xr4"
Oct 30 20:20:30.952225 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:20:30.952148    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:20:32.194064 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:32.194036    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:32.194451 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:32.194114    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:45.482773 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:45.482741    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-m8znf-jgfmg"]
Oct 30 20:20:45.483134 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:45.483092    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-m8znf-jgfmg" podUID="09961adc-4403-44ec-8c11-81938ae4fb16" containerName="insights-gathering" containerID="cri-o://5e3624625643302dc07595a1d52704a41c51446724d6b6ab46d3a3636892d3c6" gracePeriod=30
Oct 30 20:20:45.706227 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:45.706200    4471 generic.go:356] "Generic (PLEG): container finished" podID="09961adc-4403-44ec-8c11-81938ae4fb16" containerID="5e3624625643302dc07595a1d52704a41c51446724d6b6ab46d3a3636892d3c6" exitCode=2
Oct 30 20:20:45.706383 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:45.706254    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-m8znf-jgfmg" event={"ID":"09961adc-4403-44ec-8c11-81938ae4fb16","Type":"ContainerDied","Data":"5e3624625643302dc07595a1d52704a41c51446724d6b6ab46d3a3636892d3c6"}
Oct 30 20:20:45.757883 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:45.757835    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:20:45.856389 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:45.856360    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09961adc-4403-44ec-8c11-81938ae4fb16-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/09961adc-4403-44ec-8c11-81938ae4fb16-archives-path\") pod \"09961adc-4403-44ec-8c11-81938ae4fb16\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") "
Oct 30 20:20:45.856519 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:45.856423    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09961adc-4403-44ec-8c11-81938ae4fb16-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09961adc-4403-44ec-8c11-81938ae4fb16-service-ca-bundle\") pod \"09961adc-4403-44ec-8c11-81938ae4fb16\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") "
Oct 30 20:20:45.856519 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:45.856452    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09961adc-4403-44ec-8c11-81938ae4fb16-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09961adc-4403-44ec-8c11-81938ae4fb16-trusted-ca-bundle\") pod \"09961adc-4403-44ec-8c11-81938ae4fb16\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") "
Oct 30 20:20:45.856519 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:45.856494    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09961adc-4403-44ec-8c11-81938ae4fb16-kube-api-access-4m5xt\" (UniqueName: \"kubernetes.io/projected/09961adc-4403-44ec-8c11-81938ae4fb16-kube-api-access-4m5xt\") pod \"09961adc-4403-44ec-8c11-81938ae4fb16\" (UID: \"09961adc-4403-44ec-8c11-81938ae4fb16\") "
Oct 30 20:20:45.856694 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:45.856668    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09961adc-4403-44ec-8c11-81938ae4fb16-archives-path" pod "09961adc-4403-44ec-8c11-81938ae4fb16" (UID: "09961adc-4403-44ec-8c11-81938ae4fb16"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:45.856840 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:45.856814    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09961adc-4403-44ec-8c11-81938ae4fb16-service-ca-bundle" pod "09961adc-4403-44ec-8c11-81938ae4fb16" (UID: "09961adc-4403-44ec-8c11-81938ae4fb16"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:45.856898 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:45.856842    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09961adc-4403-44ec-8c11-81938ae4fb16-trusted-ca-bundle" pod "09961adc-4403-44ec-8c11-81938ae4fb16" (UID: "09961adc-4403-44ec-8c11-81938ae4fb16"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:45.858820 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:45.858797    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09961adc-4403-44ec-8c11-81938ae4fb16-kube-api-access-4m5xt" pod "09961adc-4403-44ec-8c11-81938ae4fb16" (UID: "09961adc-4403-44ec-8c11-81938ae4fb16"). InnerVolumeSpecName "kube-api-access-4m5xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:45.957193 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:45.957170    4471 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09961adc-4403-44ec-8c11-81938ae4fb16-trusted-ca-bundle\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:45.957193 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:45.957192    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m5xt\" (UniqueName: \"kubernetes.io/projected/09961adc-4403-44ec-8c11-81938ae4fb16-kube-api-access-4m5xt\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:45.957355 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:45.957202    4471 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/09961adc-4403-44ec-8c11-81938ae4fb16-archives-path\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:45.957355 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:45.957210    4471 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09961adc-4403-44ec-8c11-81938ae4fb16-service-ca-bundle\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:46.710315 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:46.710252    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-m8znf-jgfmg" event={"ID":"09961adc-4403-44ec-8c11-81938ae4fb16","Type":"ContainerDied","Data":"ad5afd8493f2d8587877e3d0b6dacc834677ef236c98caae8deb086eb44ac13a"}
Oct 30 20:20:46.710315 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:46.710274    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-m8znf-jgfmg"
Oct 30 20:20:46.710315 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:46.710304    4471 scope.go:122] "RemoveContainer" containerID="5e3624625643302dc07595a1d52704a41c51446724d6b6ab46d3a3636892d3c6"
Oct 30 20:20:46.871896 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:46.871867    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-m8znf-jgfmg"]
Oct 30 20:20:46.921042 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:46.921015    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-m8znf-jgfmg"]
Oct 30 20:20:48.428569 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:48.428534    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09961adc-4403-44ec-8c11-81938ae4fb16" path="/var/lib/kubelet/pods/09961adc-4403-44ec-8c11-81938ae4fb16/volumes"
Oct 30 20:20:52.199174 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:52.199146    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:20:52.202695 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:20:52.202670    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6d486dff54-jwr4n"
Oct 30 20:21:00.952518 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:21:00.952483    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:21:04.479080 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.479047    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"]
Oct 30 20:21:04.479533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.479260    4471 cpu_manager.go:414] "containerMap: removing container" podUID="09961adc-4403-44ec-8c11-81938ae4fb16" containerName="insights-gathering"
Oct 30 20:21:04.479533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.479270    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09961adc-4403-44ec-8c11-81938ae4fb16" containerName="insights-gathering"
Oct 30 20:21:04.479533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.479337    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="09961adc-4403-44ec-8c11-81938ae4fb16" containerName="insights-gathering"
Oct 30 20:21:04.481069 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.481051    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.581925 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.581892    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4b16608d-45cc-4a44-b345-06e3568830f3-secret-telemeter-client\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.581925 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.581927    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b16608d-45cc-4a44-b345-06e3568830f3-serving-certs-ca-bundle\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.582102 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.581946    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4b16608d-45cc-4a44-b345-06e3568830f3-federate-client-tls\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.582102 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.582038    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4b16608d-45cc-4a44-b345-06e3568830f3-telemeter-client-tls\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.582102 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.582064    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b16608d-45cc-4a44-b345-06e3568830f3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.582102 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.582080    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b16608d-45cc-4a44-b345-06e3568830f3-metrics-client-ca\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.582102 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.582095    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b16608d-45cc-4a44-b345-06e3568830f3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.582271 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.582142    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrnq7\" (UniqueName: \"kubernetes.io/projected/4b16608d-45cc-4a44-b345-06e3568830f3-kube-api-access-wrnq7\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.584523 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.584502    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 20:21:04.584639 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.584537    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 20:21:04.584724 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.584705    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 20:21:04.584794 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.584709    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-hd47v\""
Oct 30 20:21:04.584997 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.584979    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 20:21:04.585049 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.584979    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 20:21:04.585540 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.585519    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"]
Oct 30 20:21:04.635194 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.635167    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 20:21:04.682632 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.682606    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4b16608d-45cc-4a44-b345-06e3568830f3-secret-telemeter-client\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.682726 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.682638    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b16608d-45cc-4a44-b345-06e3568830f3-serving-certs-ca-bundle\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.682726 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.682658    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4b16608d-45cc-4a44-b345-06e3568830f3-federate-client-tls\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.682726 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.682684    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4b16608d-45cc-4a44-b345-06e3568830f3-telemeter-client-tls\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.682876 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.682744    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b16608d-45cc-4a44-b345-06e3568830f3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.682876 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.682772    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b16608d-45cc-4a44-b345-06e3568830f3-metrics-client-ca\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.682876 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.682796    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b16608d-45cc-4a44-b345-06e3568830f3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.682876 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.682818    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrnq7\" (UniqueName: \"kubernetes.io/projected/4b16608d-45cc-4a44-b345-06e3568830f3-kube-api-access-wrnq7\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.683565 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.683543    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b16608d-45cc-4a44-b345-06e3568830f3-metrics-client-ca\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.683713 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.683584    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b16608d-45cc-4a44-b345-06e3568830f3-serving-certs-ca-bundle\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.684004 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.683975    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b16608d-45cc-4a44-b345-06e3568830f3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.685439 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.685416    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4b16608d-45cc-4a44-b345-06e3568830f3-secret-telemeter-client\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.685527 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.685484    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4b16608d-45cc-4a44-b345-06e3568830f3-federate-client-tls\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.685656 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.685635    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4b16608d-45cc-4a44-b345-06e3568830f3-telemeter-client-tls\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.685701 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.685685    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b16608d-45cc-4a44-b345-06e3568830f3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.744893 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.744842    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrnq7\" (UniqueName: \"kubernetes.io/projected/4b16608d-45cc-4a44-b345-06e3568830f3-kube-api-access-wrnq7\") pod \"telemeter-client-7c8fbc85f8-pdmtt\" (UID: \"4b16608d-45cc-4a44-b345-06e3568830f3\") " pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:04.791540 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:04.791519    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"
Oct 30 20:21:05.208878 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:05.208855    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt"]
Oct 30 20:21:05.211399 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:21:05.211369    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b16608d_45cc_4a44_b345_06e3568830f3.slice/crio-b6d80ae8b0459b310a3fd8d3a04f6ed33efbee3bad6183b74ed6357a14a37eba WatchSource:0}: Error finding container b6d80ae8b0459b310a3fd8d3a04f6ed33efbee3bad6183b74ed6357a14a37eba: Status 404 returned error can't find the container with id b6d80ae8b0459b310a3fd8d3a04f6ed33efbee3bad6183b74ed6357a14a37eba
Oct 30 20:21:05.772818 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:05.772787    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt" event={"ID":"4b16608d-45cc-4a44-b345-06e3568830f3","Type":"ContainerStarted","Data":"b6d80ae8b0459b310a3fd8d3a04f6ed33efbee3bad6183b74ed6357a14a37eba"}
Oct 30 20:21:07.784897 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:07.784859    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt" event={"ID":"4b16608d-45cc-4a44-b345-06e3568830f3","Type":"ContainerStarted","Data":"21138b7fe8d9b788333c6bec8d1d3a60c26d29224bd8073b4a840dc579e665f0"}
Oct 30 20:21:08.790594 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:08.790511    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt" event={"ID":"4b16608d-45cc-4a44-b345-06e3568830f3","Type":"ContainerStarted","Data":"bb1edb0c5c1ccf14ec5c864c64f3465f8b937cb842df652ad31aa95b7ab39be8"}
Oct 30 20:21:08.790594 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:08.790567    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt" event={"ID":"4b16608d-45cc-4a44-b345-06e3568830f3","Type":"ContainerStarted","Data":"a75adcc36d5b6b777c523a26a3c87f72386a39d05732bc0bdd33b015b21963cf"}
Oct 30 20:21:09.009591 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:21:09.009544    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7c8fbc85f8-pdmtt" podStartSLOduration=1.7921757409999999 podStartE2EDuration="5.00953086s" totalImagesPullingTime="3.217355119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:08.95957418 +0000 UTC m=+88.638202795" watchObservedRunningTime="2026-10-30 20:21:09.00953086 +0000 UTC m=+88.688159473"
Oct 30 20:21:30.952185 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:21:30.952147    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:22:00.952114 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:22:00.952040    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:22:30.952920 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:22:30.952881    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:23:00.952401 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:23:00.952364    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:23:30.952554 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:23:30.952462    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:24:00.953696 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:24:00.953537    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:24:14.393882 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.393846    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz"]
Oct 30 20:24:14.397184 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.397166    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz"
Oct 30 20:24:14.499676 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.499645    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-rbgp6\""
Oct 30 20:24:14.499813 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.499698    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 20:24:14.499813 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.499723    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 20:24:14.500596 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.500579    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz"]
Oct 30 20:24:14.509966 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.509947    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/074a3d37-2cb2-4eec-9ca3-8ec6b2636708-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz\" (UID: \"074a3d37-2cb2-4eec-9ca3-8ec6b2636708\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz"
Oct 30 20:24:14.510059 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.509973    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/074a3d37-2cb2-4eec-9ca3-8ec6b2636708-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz\" (UID: \"074a3d37-2cb2-4eec-9ca3-8ec6b2636708\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz"
Oct 30 20:24:14.510059 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.510003    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vrdl\" (UniqueName: \"kubernetes.io/projected/074a3d37-2cb2-4eec-9ca3-8ec6b2636708-kube-api-access-2vrdl\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz\" (UID: \"074a3d37-2cb2-4eec-9ca3-8ec6b2636708\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz"
Oct 30 20:24:14.550803 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.550776    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59"]
Oct 30 20:24:14.553885 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.553871    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59"
Oct 30 20:24:14.597777 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.597754    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59"]
Oct 30 20:24:14.610422 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.610402    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e537731c-677b-49ad-b92d-9fb021c497eb-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59\" (UID: \"e537731c-677b-49ad-b92d-9fb021c497eb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59"
Oct 30 20:24:14.610546 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.610465    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e537731c-677b-49ad-b92d-9fb021c497eb-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59\" (UID: \"e537731c-677b-49ad-b92d-9fb021c497eb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59"
Oct 30 20:24:14.610546 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.610530    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/074a3d37-2cb2-4eec-9ca3-8ec6b2636708-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz\" (UID: \"074a3d37-2cb2-4eec-9ca3-8ec6b2636708\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz"
Oct 30 20:24:14.610624 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.610553    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/074a3d37-2cb2-4eec-9ca3-8ec6b2636708-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz\" (UID: \"074a3d37-2cb2-4eec-9ca3-8ec6b2636708\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz"
Oct 30 20:24:14.610624 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.610591    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vrdl\" (UniqueName: \"kubernetes.io/projected/074a3d37-2cb2-4eec-9ca3-8ec6b2636708-kube-api-access-2vrdl\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz\" (UID: \"074a3d37-2cb2-4eec-9ca3-8ec6b2636708\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz"
Oct 30 20:24:14.610624 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.610614    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcj54\" (UniqueName: \"kubernetes.io/projected/e537731c-677b-49ad-b92d-9fb021c497eb-kube-api-access-vcj54\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59\" (UID: \"e537731c-677b-49ad-b92d-9fb021c497eb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59"
Oct 30 20:24:14.610906 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.610886    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/074a3d37-2cb2-4eec-9ca3-8ec6b2636708-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz\" (UID: \"074a3d37-2cb2-4eec-9ca3-8ec6b2636708\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz"
Oct 30 20:24:14.610974 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.610922    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/074a3d37-2cb2-4eec-9ca3-8ec6b2636708-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz\" (UID: \"074a3d37-2cb2-4eec-9ca3-8ec6b2636708\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz"
Oct 30 20:24:14.669530 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.669485    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vrdl\" (UniqueName: \"kubernetes.io/projected/074a3d37-2cb2-4eec-9ca3-8ec6b2636708-kube-api-access-2vrdl\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz\" (UID: \"074a3d37-2cb2-4eec-9ca3-8ec6b2636708\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz"
Oct 30 20:24:14.671455 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.671438    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-2rksq\""
Oct 30 20:24:14.711366 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.711316    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz"
Oct 30 20:24:14.711462 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.711391    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e537731c-677b-49ad-b92d-9fb021c497eb-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59\" (UID: \"e537731c-677b-49ad-b92d-9fb021c497eb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59"
Oct 30 20:24:14.711462 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.711433    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcj54\" (UniqueName: \"kubernetes.io/projected/e537731c-677b-49ad-b92d-9fb021c497eb-kube-api-access-vcj54\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59\" (UID: \"e537731c-677b-49ad-b92d-9fb021c497eb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59"
Oct 30 20:24:14.711562 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.711463    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e537731c-677b-49ad-b92d-9fb021c497eb-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59\" (UID: \"e537731c-677b-49ad-b92d-9fb021c497eb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59"
Oct 30 20:24:14.711702 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.711686    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e537731c-677b-49ad-b92d-9fb021c497eb-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59\" (UID: \"e537731c-677b-49ad-b92d-9fb021c497eb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59"
Oct 30 20:24:14.711777 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.711762    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e537731c-677b-49ad-b92d-9fb021c497eb-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59\" (UID: \"e537731c-677b-49ad-b92d-9fb021c497eb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59"
Oct 30 20:24:14.768480 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.768450    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcj54\" (UniqueName: \"kubernetes.io/projected/e537731c-677b-49ad-b92d-9fb021c497eb-kube-api-access-vcj54\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59\" (UID: \"e537731c-677b-49ad-b92d-9fb021c497eb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59"
Oct 30 20:24:14.779872 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.779497    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"]
Oct 30 20:24:14.783139 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.783118    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:14.812385 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.812362    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6357c4ef-e469-4a75-b456-71167a77d985-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lc888\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:14.812494 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.812466    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6357c4ef-e469-4a75-b456-71167a77d985-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lc888\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:14.812555 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.812508    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6357c4ef-e469-4a75-b456-71167a77d985-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lc888\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:14.812609 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.812583    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6357c4ef-e469-4a75-b456-71167a77d985-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lc888\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:14.812660 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.812618    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6357c4ef-e469-4a75-b456-71167a77d985-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lc888\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:14.812660 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.812643    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6357c4ef-e469-4a75-b456-71167a77d985-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lc888\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:14.812739 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.812719    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cptkt\" (UniqueName: \"kubernetes.io/projected/6357c4ef-e469-4a75-b456-71167a77d985-kube-api-access-cptkt\") pod \"istiod-openshift-gateway-7876bfcb74-lc888\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:14.841975 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.841952    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 20:24:14.841975 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.841959    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 20:24:14.841975 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.841974    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jtvwq\""
Oct 30 20:24:14.842430 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.842414    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 20:24:14.842495 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.842418    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 20:24:14.842495 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.842457    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 20:24:14.842495 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.842469    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 20:24:14.847128 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.847054    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"]
Oct 30 20:24:14.874049 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.874030    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59"
Oct 30 20:24:14.913917 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.913885    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6357c4ef-e469-4a75-b456-71167a77d985-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lc888\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:14.914072 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.913937    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6357c4ef-e469-4a75-b456-71167a77d985-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lc888\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:14.914072 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.913970    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6357c4ef-e469-4a75-b456-71167a77d985-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lc888\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:14.914072 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.913992    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6357c4ef-e469-4a75-b456-71167a77d985-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lc888\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:14.914072 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.914011    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6357c4ef-e469-4a75-b456-71167a77d985-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lc888\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:14.914072 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.914033    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6357c4ef-e469-4a75-b456-71167a77d985-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lc888\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:14.914072 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.914061    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cptkt\" (UniqueName: \"kubernetes.io/projected/6357c4ef-e469-4a75-b456-71167a77d985-kube-api-access-cptkt\") pod \"istiod-openshift-gateway-7876bfcb74-lc888\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:14.914572 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.914547    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6357c4ef-e469-4a75-b456-71167a77d985-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lc888\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:14.916532 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.916507    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6357c4ef-e469-4a75-b456-71167a77d985-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lc888\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:14.916622 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.916512    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6357c4ef-e469-4a75-b456-71167a77d985-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lc888\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:14.916622 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.916609    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6357c4ef-e469-4a75-b456-71167a77d985-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lc888\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:14.916735 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.916721    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6357c4ef-e469-4a75-b456-71167a77d985-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lc888\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:14.970235 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.970209    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6357c4ef-e469-4a75-b456-71167a77d985-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lc888\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:14.970893 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:14.970874    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cptkt\" (UniqueName: \"kubernetes.io/projected/6357c4ef-e469-4a75-b456-71167a77d985-kube-api-access-cptkt\") pod \"istiod-openshift-gateway-7876bfcb74-lc888\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:15.082068 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:15.082040    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz"]
Oct 30 20:24:15.084976 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:24:15.084952    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod074a3d37_2cb2_4eec_9ca3_8ec6b2636708.slice/crio-984aeebff0362dd19bbd98d0d883ebb603b99d382fe9289c14bacca01e63f761 WatchSource:0}: Error finding container 984aeebff0362dd19bbd98d0d883ebb603b99d382fe9289c14bacca01e63f761: Status 404 returned error can't find the container with id 984aeebff0362dd19bbd98d0d883ebb603b99d382fe9289c14bacca01e63f761
Oct 30 20:24:15.094427 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:15.094409    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:15.241533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:15.241465    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59"]
Oct 30 20:24:15.244164 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:24:15.244134    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode537731c_677b_49ad_b92d_9fb021c497eb.slice/crio-fb0ac8e3d7e83584013cfb7dbc60cd827655e0ca1c2e5085c276b6c152e81c30 WatchSource:0}: Error finding container fb0ac8e3d7e83584013cfb7dbc60cd827655e0ca1c2e5085c276b6c152e81c30: Status 404 returned error can't find the container with id fb0ac8e3d7e83584013cfb7dbc60cd827655e0ca1c2e5085c276b6c152e81c30
Oct 30 20:24:15.345008 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:15.344981    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz" event={"ID":"074a3d37-2cb2-4eec-9ca3-8ec6b2636708","Type":"ContainerStarted","Data":"984aeebff0362dd19bbd98d0d883ebb603b99d382fe9289c14bacca01e63f761"}
Oct 30 20:24:15.346159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:15.346132    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59" event={"ID":"e537731c-677b-49ad-b92d-9fb021c497eb","Type":"ContainerStarted","Data":"fb0ac8e3d7e83584013cfb7dbc60cd827655e0ca1c2e5085c276b6c152e81c30"}
Oct 30 20:24:15.463941 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:15.463923    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"]
Oct 30 20:24:15.466279 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:24:15.466257    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6357c4ef_e469_4a75_b456_71167a77d985.slice/crio-413b6026f6a3996cca15c72242fa303e24c6da486b27b4e9570125638f88ac91 WatchSource:0}: Error finding container 413b6026f6a3996cca15c72242fa303e24c6da486b27b4e9570125638f88ac91: Status 404 returned error can't find the container with id 413b6026f6a3996cca15c72242fa303e24c6da486b27b4e9570125638f88ac91
Oct 30 20:24:16.351729 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:16.351690    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888" event={"ID":"6357c4ef-e469-4a75-b456-71167a77d985","Type":"ContainerStarted","Data":"413b6026f6a3996cca15c72242fa303e24c6da486b27b4e9570125638f88ac91"}
Oct 30 20:24:19.559313 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:19.559281    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:24:19.559695 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:19.559363    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:24:20.376677 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:20.376625    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888" event={"ID":"6357c4ef-e469-4a75-b456-71167a77d985","Type":"ContainerStarted","Data":"290f24b4c175d29e95432dea6e62b2bc4bf6734f05756df458188491d7afab04"}
Oct 30 20:24:20.377026 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:20.376968    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:20.378696 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:20.378659    4471 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-lc888 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:24:20.378822 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:20.378737    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888" podUID="6357c4ef-e469-4a75-b456-71167a77d985" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:24:20.581972 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:20.581910    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888" podStartSLOduration=2.491511726 podStartE2EDuration="6.581892899s" totalImagesPullingTime="4.090381173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:20.580632853 +0000 UTC m=+280.259261469" watchObservedRunningTime="2026-10-30 20:24:20.581892899 +0000 UTC m=+280.260521514"
Oct 30 20:24:21.381881 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:21.381849    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:24:22.386000 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:22.385966    4471 generic.go:356] "Generic (PLEG): container finished" podID="e537731c-677b-49ad-b92d-9fb021c497eb" containerID="c31df3eef7d00e35dcf2ad21e828330e99f268f3da618c681d27080eacc8da25" exitCode=0
Oct 30 20:24:22.386420 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:22.386061    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59" event={"ID":"e537731c-677b-49ad-b92d-9fb021c497eb","Type":"ContainerDied","Data":"c31df3eef7d00e35dcf2ad21e828330e99f268f3da618c681d27080eacc8da25"}
Oct 30 20:24:23.390711 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:23.390662    4471 generic.go:356] "Generic (PLEG): container finished" podID="074a3d37-2cb2-4eec-9ca3-8ec6b2636708" containerID="7dbaf3a4b9760ce196d0f710009dea942c72aacd3443adb612170a3f42ab290f" exitCode=0
Oct 30 20:24:23.390990 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:23.390742    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz" event={"ID":"074a3d37-2cb2-4eec-9ca3-8ec6b2636708","Type":"ContainerDied","Data":"7dbaf3a4b9760ce196d0f710009dea942c72aacd3443adb612170a3f42ab290f"}
Oct 30 20:24:24.395514 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:24.395477    4471 generic.go:356] "Generic (PLEG): container finished" podID="e537731c-677b-49ad-b92d-9fb021c497eb" containerID="bbb850a8a419f6adcc8323ceb662ca947d427084e3a7f6cfe889ef2ed9e56e36" exitCode=0
Oct 30 20:24:24.395970 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:24.395540    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59" event={"ID":"e537731c-677b-49ad-b92d-9fb021c497eb","Type":"ContainerDied","Data":"bbb850a8a419f6adcc8323ceb662ca947d427084e3a7f6cfe889ef2ed9e56e36"}
Oct 30 20:24:27.409059 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:27.409022    4471 generic.go:356] "Generic (PLEG): container finished" podID="074a3d37-2cb2-4eec-9ca3-8ec6b2636708" containerID="9247a1780ab6411adba65313c66c2ddd3a12a7e333550c648477f6413680a54f" exitCode=0
Oct 30 20:24:27.409590 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:27.409086    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz" event={"ID":"074a3d37-2cb2-4eec-9ca3-8ec6b2636708","Type":"ContainerDied","Data":"9247a1780ab6411adba65313c66c2ddd3a12a7e333550c648477f6413680a54f"}
Oct 30 20:24:30.966117 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:24:30.966049    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:24:32.428116 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:32.428083    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59" event={"ID":"e537731c-677b-49ad-b92d-9fb021c497eb","Type":"ContainerDied","Data":"0f8ff2a3840e038556fd2e8e16d99ee168b7f5b5c9d53bdf834708f5adcd751d"}
Oct 30 20:24:32.428530 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:32.428082    4471 generic.go:356] "Generic (PLEG): container finished" podID="e537731c-677b-49ad-b92d-9fb021c497eb" containerID="0f8ff2a3840e038556fd2e8e16d99ee168b7f5b5c9d53bdf834708f5adcd751d" exitCode=0
Oct 30 20:24:32.429970 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:32.429950    4471 generic.go:356] "Generic (PLEG): container finished" podID="074a3d37-2cb2-4eec-9ca3-8ec6b2636708" containerID="4f5d79c41d859db9d300431ceab733110c4f0ab1f8ad619d2ed7a3b86c6da956" exitCode=0
Oct 30 20:24:32.430076 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:32.429980    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz" event={"ID":"074a3d37-2cb2-4eec-9ca3-8ec6b2636708","Type":"ContainerDied","Data":"4f5d79c41d859db9d300431ceab733110c4f0ab1f8ad619d2ed7a3b86c6da956"}
Oct 30 20:24:33.575282 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:33.575261    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59"
Oct 30 20:24:33.578229 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:33.578211    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz"
Oct 30 20:24:33.665772 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:33.665747    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/074a3d37-2cb2-4eec-9ca3-8ec6b2636708-util\" (UniqueName: \"kubernetes.io/empty-dir/074a3d37-2cb2-4eec-9ca3-8ec6b2636708-util\") pod \"074a3d37-2cb2-4eec-9ca3-8ec6b2636708\" (UID: \"074a3d37-2cb2-4eec-9ca3-8ec6b2636708\") "
Oct 30 20:24:33.665871 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:33.665784    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e537731c-677b-49ad-b92d-9fb021c497eb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e537731c-677b-49ad-b92d-9fb021c497eb-bundle\") pod \"e537731c-677b-49ad-b92d-9fb021c497eb\" (UID: \"e537731c-677b-49ad-b92d-9fb021c497eb\") "
Oct 30 20:24:33.665871 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:33.665824    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/074a3d37-2cb2-4eec-9ca3-8ec6b2636708-kube-api-access-2vrdl\" (UniqueName: \"kubernetes.io/projected/074a3d37-2cb2-4eec-9ca3-8ec6b2636708-kube-api-access-2vrdl\") pod \"074a3d37-2cb2-4eec-9ca3-8ec6b2636708\" (UID: \"074a3d37-2cb2-4eec-9ca3-8ec6b2636708\") "
Oct 30 20:24:33.665871 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:33.665849    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e537731c-677b-49ad-b92d-9fb021c497eb-kube-api-access-vcj54\" (UniqueName: \"kubernetes.io/projected/e537731c-677b-49ad-b92d-9fb021c497eb-kube-api-access-vcj54\") pod \"e537731c-677b-49ad-b92d-9fb021c497eb\" (UID: \"e537731c-677b-49ad-b92d-9fb021c497eb\") "
Oct 30 20:24:33.665871 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:33.665867    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e537731c-677b-49ad-b92d-9fb021c497eb-util\" (UniqueName: \"kubernetes.io/empty-dir/e537731c-677b-49ad-b92d-9fb021c497eb-util\") pod \"e537731c-677b-49ad-b92d-9fb021c497eb\" (UID: \"e537731c-677b-49ad-b92d-9fb021c497eb\") "
Oct 30 20:24:33.666050 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:33.665899    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/074a3d37-2cb2-4eec-9ca3-8ec6b2636708-bundle\" (UniqueName: \"kubernetes.io/empty-dir/074a3d37-2cb2-4eec-9ca3-8ec6b2636708-bundle\") pod \"074a3d37-2cb2-4eec-9ca3-8ec6b2636708\" (UID: \"074a3d37-2cb2-4eec-9ca3-8ec6b2636708\") "
Oct 30 20:24:33.666369 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:33.666344    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/074a3d37-2cb2-4eec-9ca3-8ec6b2636708-bundle" pod "074a3d37-2cb2-4eec-9ca3-8ec6b2636708" (UID: "074a3d37-2cb2-4eec-9ca3-8ec6b2636708"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:33.666491 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:33.666466    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e537731c-677b-49ad-b92d-9fb021c497eb-bundle" pod "e537731c-677b-49ad-b92d-9fb021c497eb" (UID: "e537731c-677b-49ad-b92d-9fb021c497eb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:33.668006 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:33.667987    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074a3d37-2cb2-4eec-9ca3-8ec6b2636708-kube-api-access-2vrdl" pod "074a3d37-2cb2-4eec-9ca3-8ec6b2636708" (UID: "074a3d37-2cb2-4eec-9ca3-8ec6b2636708"). InnerVolumeSpecName "kube-api-access-2vrdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:33.668270 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:33.668255    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e537731c-677b-49ad-b92d-9fb021c497eb-kube-api-access-vcj54" pod "e537731c-677b-49ad-b92d-9fb021c497eb" (UID: "e537731c-677b-49ad-b92d-9fb021c497eb"). InnerVolumeSpecName "kube-api-access-vcj54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:33.669974 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:33.669951    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e537731c-677b-49ad-b92d-9fb021c497eb-util" pod "e537731c-677b-49ad-b92d-9fb021c497eb" (UID: "e537731c-677b-49ad-b92d-9fb021c497eb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:33.670713 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:33.670694    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/074a3d37-2cb2-4eec-9ca3-8ec6b2636708-util" pod "074a3d37-2cb2-4eec-9ca3-8ec6b2636708" (UID: "074a3d37-2cb2-4eec-9ca3-8ec6b2636708"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:33.766608 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:33.766553    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vrdl\" (UniqueName: \"kubernetes.io/projected/074a3d37-2cb2-4eec-9ca3-8ec6b2636708-kube-api-access-2vrdl\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:33.766608 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:33.766576    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcj54\" (UniqueName: \"kubernetes.io/projected/e537731c-677b-49ad-b92d-9fb021c497eb-kube-api-access-vcj54\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:33.766608 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:33.766585    4471 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e537731c-677b-49ad-b92d-9fb021c497eb-util\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:33.766608 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:33.766593    4471 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/074a3d37-2cb2-4eec-9ca3-8ec6b2636708-bundle\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:33.766608 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:33.766601    4471 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/074a3d37-2cb2-4eec-9ca3-8ec6b2636708-util\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:33.766608 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:33.766608    4471 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e537731c-677b-49ad-b92d-9fb021c497eb-bundle\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:34.438170 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:34.438136    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz"
Oct 30 20:24:34.438170 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:34.438164    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644l74jz" event={"ID":"074a3d37-2cb2-4eec-9ca3-8ec6b2636708","Type":"ContainerDied","Data":"984aeebff0362dd19bbd98d0d883ebb603b99d382fe9289c14bacca01e63f761"}
Oct 30 20:24:34.438459 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:34.438210    4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="984aeebff0362dd19bbd98d0d883ebb603b99d382fe9289c14bacca01e63f761"
Oct 30 20:24:34.440250 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:34.440224    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59" event={"ID":"e537731c-677b-49ad-b92d-9fb021c497eb","Type":"ContainerDied","Data":"fb0ac8e3d7e83584013cfb7dbc60cd827655e0ca1c2e5085c276b6c152e81c30"}
Oct 30 20:24:34.440386 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:34.440253    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t2l59"
Oct 30 20:24:34.440386 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:24:34.440265    4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb0ac8e3d7e83584013cfb7dbc60cd827655e0ca1c2e5085c276b6c152e81c30"
Oct 30 20:25:00.952174 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:25:00.952090    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:25:08.029679 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.029645    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"]
Oct 30 20:25:08.030141 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.029909    4471 cpu_manager.go:414] "containerMap: removing container" podUID="074a3d37-2cb2-4eec-9ca3-8ec6b2636708" containerName="pull"
Oct 30 20:25:08.030141 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.029919    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="074a3d37-2cb2-4eec-9ca3-8ec6b2636708" containerName="pull"
Oct 30 20:25:08.030141 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.029926    4471 cpu_manager.go:414] "containerMap: removing container" podUID="e537731c-677b-49ad-b92d-9fb021c497eb" containerName="extract"
Oct 30 20:25:08.030141 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.029931    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e537731c-677b-49ad-b92d-9fb021c497eb" containerName="extract"
Oct 30 20:25:08.030141 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.029938    4471 cpu_manager.go:414] "containerMap: removing container" podUID="e537731c-677b-49ad-b92d-9fb021c497eb" containerName="util"
Oct 30 20:25:08.030141 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.029942    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e537731c-677b-49ad-b92d-9fb021c497eb" containerName="util"
Oct 30 20:25:08.030141 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.029948    4471 cpu_manager.go:414] "containerMap: removing container" podUID="074a3d37-2cb2-4eec-9ca3-8ec6b2636708" containerName="util"
Oct 30 20:25:08.030141 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.029952    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="074a3d37-2cb2-4eec-9ca3-8ec6b2636708" containerName="util"
Oct 30 20:25:08.030141 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.029957    4471 cpu_manager.go:414] "containerMap: removing container" podUID="074a3d37-2cb2-4eec-9ca3-8ec6b2636708" containerName="extract"
Oct 30 20:25:08.030141 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.029961    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="074a3d37-2cb2-4eec-9ca3-8ec6b2636708" containerName="extract"
Oct 30 20:25:08.030141 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.029971    4471 cpu_manager.go:414] "containerMap: removing container" podUID="e537731c-677b-49ad-b92d-9fb021c497eb" containerName="pull"
Oct 30 20:25:08.030141 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.029975    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e537731c-677b-49ad-b92d-9fb021c497eb" containerName="pull"
Oct 30 20:25:08.030141 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.030012    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="e537731c-677b-49ad-b92d-9fb021c497eb" containerName="extract"
Oct 30 20:25:08.030141 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.030019    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="074a3d37-2cb2-4eec-9ca3-8ec6b2636708" containerName="extract"
Oct 30 20:25:08.031688 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.031673    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"
Oct 30 20:25:08.087693 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.087645    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 20:25:08.087839 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.087768    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 20:25:08.137896 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.137866    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"]
Oct 30 20:25:08.177267 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.177241    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 20:25:08.177686 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.177669    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 20:25:08.177879 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.177862    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 20:25:08.177916 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.177862    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-trbjm\""
Oct 30 20:25:08.198348 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.198309    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5da6bbde-6707-475f-bfe6-646ad004845a-manifests\") pod \"opendatahub-operator-controller-manager-585d4dbb69-lfwcv\" (UID: \"5da6bbde-6707-475f-bfe6-646ad004845a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"
Oct 30 20:25:08.198348 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.198350    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fstz9\" (UniqueName: \"kubernetes.io/projected/5da6bbde-6707-475f-bfe6-646ad004845a-kube-api-access-fstz9\") pod \"opendatahub-operator-controller-manager-585d4dbb69-lfwcv\" (UID: \"5da6bbde-6707-475f-bfe6-646ad004845a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"
Oct 30 20:25:08.198523 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.198375    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5da6bbde-6707-475f-bfe6-646ad004845a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-lfwcv\" (UID: \"5da6bbde-6707-475f-bfe6-646ad004845a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"
Oct 30 20:25:08.198523 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.198510    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5da6bbde-6707-475f-bfe6-646ad004845a-tmp\") pod \"opendatahub-operator-controller-manager-585d4dbb69-lfwcv\" (UID: \"5da6bbde-6707-475f-bfe6-646ad004845a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"
Oct 30 20:25:08.198595 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.198540    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5da6bbde-6707-475f-bfe6-646ad004845a-webhook-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-lfwcv\" (UID: \"5da6bbde-6707-475f-bfe6-646ad004845a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"
Oct 30 20:25:08.198595 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.198562    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5da6bbde-6707-475f-bfe6-646ad004845a-metrics-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-lfwcv\" (UID: \"5da6bbde-6707-475f-bfe6-646ad004845a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"
Oct 30 20:25:08.299069 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.298999    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5da6bbde-6707-475f-bfe6-646ad004845a-manifests\") pod \"opendatahub-operator-controller-manager-585d4dbb69-lfwcv\" (UID: \"5da6bbde-6707-475f-bfe6-646ad004845a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"
Oct 30 20:25:08.299069 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.299034    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fstz9\" (UniqueName: \"kubernetes.io/projected/5da6bbde-6707-475f-bfe6-646ad004845a-kube-api-access-fstz9\") pod \"opendatahub-operator-controller-manager-585d4dbb69-lfwcv\" (UID: \"5da6bbde-6707-475f-bfe6-646ad004845a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"
Oct 30 20:25:08.299265 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.299072    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5da6bbde-6707-475f-bfe6-646ad004845a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-lfwcv\" (UID: \"5da6bbde-6707-475f-bfe6-646ad004845a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"
Oct 30 20:25:08.299265 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.299133    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5da6bbde-6707-475f-bfe6-646ad004845a-tmp\") pod \"opendatahub-operator-controller-manager-585d4dbb69-lfwcv\" (UID: \"5da6bbde-6707-475f-bfe6-646ad004845a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"
Oct 30 20:25:08.299265 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.299170    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5da6bbde-6707-475f-bfe6-646ad004845a-webhook-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-lfwcv\" (UID: \"5da6bbde-6707-475f-bfe6-646ad004845a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"
Oct 30 20:25:08.299265 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.299197    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5da6bbde-6707-475f-bfe6-646ad004845a-metrics-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-lfwcv\" (UID: \"5da6bbde-6707-475f-bfe6-646ad004845a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"
Oct 30 20:25:08.299516 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.299490    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5da6bbde-6707-475f-bfe6-646ad004845a-manifests\") pod \"opendatahub-operator-controller-manager-585d4dbb69-lfwcv\" (UID: \"5da6bbde-6707-475f-bfe6-646ad004845a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"
Oct 30 20:25:08.301681 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.301653    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5da6bbde-6707-475f-bfe6-646ad004845a-tmp\") pod \"opendatahub-operator-controller-manager-585d4dbb69-lfwcv\" (UID: \"5da6bbde-6707-475f-bfe6-646ad004845a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"
Oct 30 20:25:08.301792 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.301681    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5da6bbde-6707-475f-bfe6-646ad004845a-metrics-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-lfwcv\" (UID: \"5da6bbde-6707-475f-bfe6-646ad004845a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"
Oct 30 20:25:08.301870 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.301848    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5da6bbde-6707-475f-bfe6-646ad004845a-webhook-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-lfwcv\" (UID: \"5da6bbde-6707-475f-bfe6-646ad004845a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"
Oct 30 20:25:08.301913 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.301848    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5da6bbde-6707-475f-bfe6-646ad004845a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-585d4dbb69-lfwcv\" (UID: \"5da6bbde-6707-475f-bfe6-646ad004845a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"
Oct 30 20:25:08.355856 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.355829    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fstz9\" (UniqueName: \"kubernetes.io/projected/5da6bbde-6707-475f-bfe6-646ad004845a-kube-api-access-fstz9\") pod \"opendatahub-operator-controller-manager-585d4dbb69-lfwcv\" (UID: \"5da6bbde-6707-475f-bfe6-646ad004845a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"
Oct 30 20:25:08.644640 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:08.644611    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"
Oct 30 20:25:09.237601 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:09.237567    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"]
Oct 30 20:25:09.262541 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:25:09.262516    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da6bbde_6707_475f_bfe6_646ad004845a.slice/crio-521b8d44f804917732dd58a5563cf544055760521c965be52f990451a2754c7b WatchSource:0}: Error finding container 521b8d44f804917732dd58a5563cf544055760521c965be52f990451a2754c7b: Status 404 returned error can't find the container with id 521b8d44f804917732dd58a5563cf544055760521c965be52f990451a2754c7b
Oct 30 20:25:09.264015 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:09.263998    4471 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:25:09.555927 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:09.555846    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv" event={"ID":"5da6bbde-6707-475f-bfe6-646ad004845a","Type":"ContainerStarted","Data":"521b8d44f804917732dd58a5563cf544055760521c965be52f990451a2754c7b"}
Oct 30 20:25:12.567121 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:12.567087    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv" event={"ID":"5da6bbde-6707-475f-bfe6-646ad004845a","Type":"ContainerStarted","Data":"027b47c087962176d63025c984d18ee1937026fefe2d50a83d1e599a057d92d1"}
Oct 30 20:25:13.572210 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:13.572167    4471 generic.go:356] "Generic (PLEG): container finished" podID="5da6bbde-6707-475f-bfe6-646ad004845a" containerID="027b47c087962176d63025c984d18ee1937026fefe2d50a83d1e599a057d92d1" exitCode=0
Oct 30 20:25:13.572689 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:13.572259    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv" event={"ID":"5da6bbde-6707-475f-bfe6-646ad004845a","Type":"ContainerDied","Data":"027b47c087962176d63025c984d18ee1937026fefe2d50a83d1e599a057d92d1"}
Oct 30 20:25:14.578601 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:14.578566    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv" event={"ID":"5da6bbde-6707-475f-bfe6-646ad004845a","Type":"ContainerStarted","Data":"65273c24dbc08643b601e5566d3649c0647a15686aac7ba04c0a1fb728c4f39b"}
Oct 30 20:25:14.579039 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:14.578874    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"
Oct 30 20:25:14.715894 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:14.715841    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv" podStartSLOduration=3.166433498 podStartE2EDuration="7.715823813s" totalImagesPullingTime="4.145650072s" totalInitContainerRuntime="403.740243ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:14.713470875 +0000 UTC m=+334.392099493" watchObservedRunningTime="2026-10-30 20:25:14.715823813 +0000 UTC m=+334.394452432"
Oct 30 20:25:25.584923 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:25.584888    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-585d4dbb69-lfwcv"
Oct 30 20:25:30.952674 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:25:30.952638    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:25:38.054770 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.054733    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vp269"]
Oct 30 20:25:38.056673 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.056655    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vp269"
Oct 30 20:25:38.108923 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.108895    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 20:25:38.109054 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.108958    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 20:25:38.113248 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.113228    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1d5265ac-02a8-4695-8c71-60a60115a3c6-config-file\") pod \"limitador-limitador-74dfcfb894-vp269\" (UID: \"1d5265ac-02a8-4695-8c71-60a60115a3c6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vp269"
Oct 30 20:25:38.113356 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.113269    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh2d8\" (UniqueName: \"kubernetes.io/projected/1d5265ac-02a8-4695-8c71-60a60115a3c6-kube-api-access-rh2d8\") pod \"limitador-limitador-74dfcfb894-vp269\" (UID: \"1d5265ac-02a8-4695-8c71-60a60115a3c6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vp269"
Oct 30 20:25:38.158512 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.158483    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rjp9d\""
Oct 30 20:25:38.158512 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.158492    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 20:25:38.159544 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.159522    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vp269"]
Oct 30 20:25:38.214372 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.214342    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1d5265ac-02a8-4695-8c71-60a60115a3c6-config-file\") pod \"limitador-limitador-74dfcfb894-vp269\" (UID: \"1d5265ac-02a8-4695-8c71-60a60115a3c6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vp269"
Oct 30 20:25:38.214505 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.214402    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh2d8\" (UniqueName: \"kubernetes.io/projected/1d5265ac-02a8-4695-8c71-60a60115a3c6-kube-api-access-rh2d8\") pod \"limitador-limitador-74dfcfb894-vp269\" (UID: \"1d5265ac-02a8-4695-8c71-60a60115a3c6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vp269"
Oct 30 20:25:38.214909 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.214891    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1d5265ac-02a8-4695-8c71-60a60115a3c6-config-file\") pod \"limitador-limitador-74dfcfb894-vp269\" (UID: \"1d5265ac-02a8-4695-8c71-60a60115a3c6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vp269"
Oct 30 20:25:38.278359 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.278314    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh2d8\" (UniqueName: \"kubernetes.io/projected/1d5265ac-02a8-4695-8c71-60a60115a3c6-kube-api-access-rh2d8\") pod \"limitador-limitador-74dfcfb894-vp269\" (UID: \"1d5265ac-02a8-4695-8c71-60a60115a3c6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vp269"
Oct 30 20:25:38.296083 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.296050    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vp269"]
Oct 30 20:25:38.367068 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.366998    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vp269"
Oct 30 20:25:38.596671 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.596640    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-557f979cb4-v95bs"]
Oct 30 20:25:38.598707 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.598689    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-557f979cb4-v95bs"
Oct 30 20:25:38.658711 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.658641    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-557f979cb4-v95bs"]
Oct 30 20:25:38.671255 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.671223    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8g74l\""
Oct 30 20:25:38.671452 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.671364    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 20:25:38.671532 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.671502    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 20:25:38.716909 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.716884    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6e9bfd6-b580-4857-990f-b23782215f70-tmp\") pod \"kube-auth-proxy-557f979cb4-v95bs\" (UID: \"f6e9bfd6-b580-4857-990f-b23782215f70\") " pod="openshift-ingress/kube-auth-proxy-557f979cb4-v95bs"
Oct 30 20:25:38.717067 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.716917    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6c75\" (UniqueName: \"kubernetes.io/projected/f6e9bfd6-b580-4857-990f-b23782215f70-kube-api-access-h6c75\") pod \"kube-auth-proxy-557f979cb4-v95bs\" (UID: \"f6e9bfd6-b580-4857-990f-b23782215f70\") " pod="openshift-ingress/kube-auth-proxy-557f979cb4-v95bs"
Oct 30 20:25:38.717067 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.716941    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6e9bfd6-b580-4857-990f-b23782215f70-tls-certs\") pod \"kube-auth-proxy-557f979cb4-v95bs\" (UID: \"f6e9bfd6-b580-4857-990f-b23782215f70\") " pod="openshift-ingress/kube-auth-proxy-557f979cb4-v95bs"
Oct 30 20:25:38.771716 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.771687    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vp269"]
Oct 30 20:25:38.775399 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:25:38.775359    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5265ac_02a8_4695_8c71_60a60115a3c6.slice/crio-e4641aaae0e1cb211f03effd2de25bf1a8cb874f5f8655d25e2b819cb531e34f WatchSource:0}: Error finding container e4641aaae0e1cb211f03effd2de25bf1a8cb874f5f8655d25e2b819cb531e34f: Status 404 returned error can't find the container with id e4641aaae0e1cb211f03effd2de25bf1a8cb874f5f8655d25e2b819cb531e34f
Oct 30 20:25:38.817262 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.817244    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6e9bfd6-b580-4857-990f-b23782215f70-tmp\") pod \"kube-auth-proxy-557f979cb4-v95bs\" (UID: \"f6e9bfd6-b580-4857-990f-b23782215f70\") " pod="openshift-ingress/kube-auth-proxy-557f979cb4-v95bs"
Oct 30 20:25:38.817356 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.817273    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6c75\" (UniqueName: \"kubernetes.io/projected/f6e9bfd6-b580-4857-990f-b23782215f70-kube-api-access-h6c75\") pod \"kube-auth-proxy-557f979cb4-v95bs\" (UID: \"f6e9bfd6-b580-4857-990f-b23782215f70\") " pod="openshift-ingress/kube-auth-proxy-557f979cb4-v95bs"
Oct 30 20:25:38.817356 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.817305    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6e9bfd6-b580-4857-990f-b23782215f70-tls-certs\") pod \"kube-auth-proxy-557f979cb4-v95bs\" (UID: \"f6e9bfd6-b580-4857-990f-b23782215f70\") " pod="openshift-ingress/kube-auth-proxy-557f979cb4-v95bs"
Oct 30 20:25:38.819644 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.819627    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6e9bfd6-b580-4857-990f-b23782215f70-tmp\") pod \"kube-auth-proxy-557f979cb4-v95bs\" (UID: \"f6e9bfd6-b580-4857-990f-b23782215f70\") " pod="openshift-ingress/kube-auth-proxy-557f979cb4-v95bs"
Oct 30 20:25:38.819838 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.819818    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6e9bfd6-b580-4857-990f-b23782215f70-tls-certs\") pod \"kube-auth-proxy-557f979cb4-v95bs\" (UID: \"f6e9bfd6-b580-4857-990f-b23782215f70\") " pod="openshift-ingress/kube-auth-proxy-557f979cb4-v95bs"
Oct 30 20:25:38.881481 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.881456    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6c75\" (UniqueName: \"kubernetes.io/projected/f6e9bfd6-b580-4857-990f-b23782215f70-kube-api-access-h6c75\") pod \"kube-auth-proxy-557f979cb4-v95bs\" (UID: \"f6e9bfd6-b580-4857-990f-b23782215f70\") " pod="openshift-ingress/kube-auth-proxy-557f979cb4-v95bs"
Oct 30 20:25:38.908477 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:38.908452    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-557f979cb4-v95bs"
Oct 30 20:25:39.275102 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:39.274997    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-557f979cb4-v95bs"]
Oct 30 20:25:39.277455 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:25:39.277428    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e9bfd6_b580_4857_990f_b23782215f70.slice/crio-3da03febf0f79c12840f636a4b6ae1d2524167ec069373e360d80de157730656 WatchSource:0}: Error finding container 3da03febf0f79c12840f636a4b6ae1d2524167ec069373e360d80de157730656: Status 404 returned error can't find the container with id 3da03febf0f79c12840f636a4b6ae1d2524167ec069373e360d80de157730656
Oct 30 20:25:39.664043 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:39.664009    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vp269" event={"ID":"1d5265ac-02a8-4695-8c71-60a60115a3c6","Type":"ContainerStarted","Data":"e4641aaae0e1cb211f03effd2de25bf1a8cb874f5f8655d25e2b819cb531e34f"}
Oct 30 20:25:39.665195 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:39.665168    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-557f979cb4-v95bs" event={"ID":"f6e9bfd6-b580-4857-990f-b23782215f70","Type":"ContainerStarted","Data":"3da03febf0f79c12840f636a4b6ae1d2524167ec069373e360d80de157730656"}
Oct 30 20:25:42.385856 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:42.385831    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 20:25:42.686404 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:42.686365    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vp269" event={"ID":"1d5265ac-02a8-4695-8c71-60a60115a3c6","Type":"ContainerStarted","Data":"b253694f22305d33c39bad8400dbb6020cce3f4adf314e0e644c8918387effb1"}
Oct 30 20:25:42.686617 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:42.686475    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-vp269"
Oct 30 20:25:42.687806 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:42.687783    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-557f979cb4-v95bs" event={"ID":"f6e9bfd6-b580-4857-990f-b23782215f70","Type":"ContainerStarted","Data":"1b3b870449e523585175f5ce2ca797f61fbc1ba1696372ea33aade6b5764976c"}
Oct 30 20:25:42.801070 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:42.801020    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-vp269" podStartSLOduration=1.253605765 podStartE2EDuration="4.801008262s" totalImagesPullingTime="3.547402497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:42.799745397 +0000 UTC m=+362.478374014" watchObservedRunningTime="2026-10-30 20:25:42.801008262 +0000 UTC m=+362.479636900"
Oct 30 20:25:42.913099 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:42.913052    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-557f979cb4-v95bs" podStartSLOduration=1.857933581 podStartE2EDuration="4.913039451s" totalImagesPullingTime="3.05510587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:42.912437387 +0000 UTC m=+362.591066002" watchObservedRunningTime="2026-10-30 20:25:42.913039451 +0000 UTC m=+362.591668068"
Oct 30 20:25:53.692415 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:25:53.692386    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-vp269"
Oct 30 20:26:00.952649 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:26:00.952611    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:26:04.928249 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:04.928216    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-zf85x"]
Oct 30 20:26:04.931394 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:04.931379    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zf85x"
Oct 30 20:26:04.984004 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:04.983978    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:26:05.012295 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:05.012270    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7mvm\" (UniqueName: \"kubernetes.io/projected/3ccee963-b721-45eb-b864-6355fa869970-kube-api-access-z7mvm\") pod \"postgres-7cdbf6dc44-zf85x\" (UID: \"3ccee963-b721-45eb-b864-6355fa869970\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zf85x"
Oct 30 20:26:05.012437 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:05.012316    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3ccee963-b721-45eb-b864-6355fa869970-data\") pod \"postgres-7cdbf6dc44-zf85x\" (UID: \"3ccee963-b721-45eb-b864-6355fa869970\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zf85x"
Oct 30 20:26:05.025979 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:05.025961    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:26:05.032690 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:05.032669    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 20:26:05.032775 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:05.032710    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-d7m29\""
Oct 30 20:26:05.034564 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:05.034538    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-zf85x"]
Oct 30 20:26:05.113015 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:05.112988    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7mvm\" (UniqueName: \"kubernetes.io/projected/3ccee963-b721-45eb-b864-6355fa869970-kube-api-access-z7mvm\") pod \"postgres-7cdbf6dc44-zf85x\" (UID: \"3ccee963-b721-45eb-b864-6355fa869970\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zf85x"
Oct 30 20:26:05.113160 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:05.113039    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3ccee963-b721-45eb-b864-6355fa869970-data\") pod \"postgres-7cdbf6dc44-zf85x\" (UID: \"3ccee963-b721-45eb-b864-6355fa869970\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zf85x"
Oct 30 20:26:05.113441 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:05.113422    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3ccee963-b721-45eb-b864-6355fa869970-data\") pod \"postgres-7cdbf6dc44-zf85x\" (UID: \"3ccee963-b721-45eb-b864-6355fa869970\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zf85x"
Oct 30 20:26:05.169400 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:05.169378    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7mvm\" (UniqueName: \"kubernetes.io/projected/3ccee963-b721-45eb-b864-6355fa869970-kube-api-access-z7mvm\") pod \"postgres-7cdbf6dc44-zf85x\" (UID: \"3ccee963-b721-45eb-b864-6355fa869970\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zf85x"
Oct 30 20:26:05.245372 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:05.245277    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zf85x"
Oct 30 20:26:05.610578 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:05.610504    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-zf85x"]
Oct 30 20:26:05.614075 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:26:05.614047    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ccee963_b721_45eb_b864_6355fa869970.slice/crio-daf6d213f2d299a7b9df54b6e926a1ba575fa87fedcab06d65ee09bcd916b484 WatchSource:0}: Error finding container daf6d213f2d299a7b9df54b6e926a1ba575fa87fedcab06d65ee09bcd916b484: Status 404 returned error can't find the container with id daf6d213f2d299a7b9df54b6e926a1ba575fa87fedcab06d65ee09bcd916b484
Oct 30 20:26:05.769148 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:05.769113    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zf85x" event={"ID":"3ccee963-b721-45eb-b864-6355fa869970","Type":"ContainerStarted","Data":"daf6d213f2d299a7b9df54b6e926a1ba575fa87fedcab06d65ee09bcd916b484"}
Oct 30 20:26:12.798082 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:12.798049    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zf85x" event={"ID":"3ccee963-b721-45eb-b864-6355fa869970","Type":"ContainerStarted","Data":"6318a6787905ecdaa4370575a861bcdb1f6003a7659bbae94b7fda041b8bab78"}
Oct 30 20:26:12.798467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:12.798180    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zf85x"
Oct 30 20:26:12.912140 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:12.912086    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zf85x" podStartSLOduration=2.7445191209999997 podStartE2EDuration="8.912070023s" totalImagesPullingTime="6.167550902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:12.911505915 +0000 UTC m=+392.590134529" watchObservedRunningTime="2026-10-30 20:26:12.912070023 +0000 UTC m=+392.590698645"
Oct 30 20:26:14.991214 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:14.991180    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vp269"]
Oct 30 20:26:14.991688 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:14.991382    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-vp269" podUID="1d5265ac-02a8-4695-8c71-60a60115a3c6" containerName="limitador" containerID="cri-o://b253694f22305d33c39bad8400dbb6020cce3f4adf314e0e644c8918387effb1" gracePeriod=30
Oct 30 20:26:15.520515 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:15.520491    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vp269"
Oct 30 20:26:15.597534 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:15.597468    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d5265ac-02a8-4695-8c71-60a60115a3c6-config-file\" (UniqueName: \"kubernetes.io/configmap/1d5265ac-02a8-4695-8c71-60a60115a3c6-config-file\") pod \"1d5265ac-02a8-4695-8c71-60a60115a3c6\" (UID: \"1d5265ac-02a8-4695-8c71-60a60115a3c6\") "
Oct 30 20:26:15.597643 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:15.597544    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d5265ac-02a8-4695-8c71-60a60115a3c6-kube-api-access-rh2d8\" (UniqueName: \"kubernetes.io/projected/1d5265ac-02a8-4695-8c71-60a60115a3c6-kube-api-access-rh2d8\") pod \"1d5265ac-02a8-4695-8c71-60a60115a3c6\" (UID: \"1d5265ac-02a8-4695-8c71-60a60115a3c6\") "
Oct 30 20:26:15.597808 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:15.597784    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d5265ac-02a8-4695-8c71-60a60115a3c6-config-file" pod "1d5265ac-02a8-4695-8c71-60a60115a3c6" (UID: "1d5265ac-02a8-4695-8c71-60a60115a3c6"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:15.599657 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:15.599633    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5265ac-02a8-4695-8c71-60a60115a3c6-kube-api-access-rh2d8" pod "1d5265ac-02a8-4695-8c71-60a60115a3c6" (UID: "1d5265ac-02a8-4695-8c71-60a60115a3c6"). InnerVolumeSpecName "kube-api-access-rh2d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:15.698278 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:15.698251    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh2d8\" (UniqueName: \"kubernetes.io/projected/1d5265ac-02a8-4695-8c71-60a60115a3c6-kube-api-access-rh2d8\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:15.698278 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:15.698278    4471 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1d5265ac-02a8-4695-8c71-60a60115a3c6-config-file\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:15.810824 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:15.810794    4471 generic.go:356] "Generic (PLEG): container finished" podID="1d5265ac-02a8-4695-8c71-60a60115a3c6" containerID="b253694f22305d33c39bad8400dbb6020cce3f4adf314e0e644c8918387effb1" exitCode=0
Oct 30 20:26:15.811005 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:15.810845    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vp269"
Oct 30 20:26:15.811005 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:15.810848    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vp269" event={"ID":"1d5265ac-02a8-4695-8c71-60a60115a3c6","Type":"ContainerDied","Data":"b253694f22305d33c39bad8400dbb6020cce3f4adf314e0e644c8918387effb1"}
Oct 30 20:26:15.811005 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:15.810896    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vp269" event={"ID":"1d5265ac-02a8-4695-8c71-60a60115a3c6","Type":"ContainerDied","Data":"e4641aaae0e1cb211f03effd2de25bf1a8cb874f5f8655d25e2b819cb531e34f"}
Oct 30 20:26:15.811005 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:15.810917    4471 scope.go:122] "RemoveContainer" containerID="b253694f22305d33c39bad8400dbb6020cce3f4adf314e0e644c8918387effb1"
Oct 30 20:26:15.819681 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:15.819666    4471 scope.go:122] "RemoveContainer" containerID="b253694f22305d33c39bad8400dbb6020cce3f4adf314e0e644c8918387effb1"
Oct 30 20:26:15.819985 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:26:15.819965    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b253694f22305d33c39bad8400dbb6020cce3f4adf314e0e644c8918387effb1\": container with ID starting with b253694f22305d33c39bad8400dbb6020cce3f4adf314e0e644c8918387effb1 not found: ID does not exist" containerID="b253694f22305d33c39bad8400dbb6020cce3f4adf314e0e644c8918387effb1"
Oct 30 20:26:15.820063 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:15.819995    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b253694f22305d33c39bad8400dbb6020cce3f4adf314e0e644c8918387effb1"} err="failed to get container status \"b253694f22305d33c39bad8400dbb6020cce3f4adf314e0e644c8918387effb1\": rpc error: code = NotFound desc = could not find container \"b253694f22305d33c39bad8400dbb6020cce3f4adf314e0e644c8918387effb1\": container with ID starting with b253694f22305d33c39bad8400dbb6020cce3f4adf314e0e644c8918387effb1 not found: ID does not exist"
Oct 30 20:26:15.976161 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:15.976135    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vp269"]
Oct 30 20:26:15.980029 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:15.980008    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vp269"]
Oct 30 20:26:16.429196 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:16.429165    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d5265ac-02a8-4695-8c71-60a60115a3c6" path="/var/lib/kubelet/pods/1d5265ac-02a8-4695-8c71-60a60115a3c6/volumes"
Oct 30 20:26:18.830266 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:18.830243    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zf85x"
Oct 30 20:26:25.968102 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:25.968032    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-vlf2g"]
Oct 30 20:26:25.968563 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:25.968461    4471 cpu_manager.go:414] "containerMap: removing container" podUID="1d5265ac-02a8-4695-8c71-60a60115a3c6" containerName="limitador"
Oct 30 20:26:25.968563 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:25.968480    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d5265ac-02a8-4695-8c71-60a60115a3c6" containerName="limitador"
Oct 30 20:26:25.968563 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:25.968559    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d5265ac-02a8-4695-8c71-60a60115a3c6" containerName="limitador"
Oct 30 20:26:25.971555 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:25.971537    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-vlf2g"
Oct 30 20:26:26.029944 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.029917    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 20:26:26.074301 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.074275    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zrcf\" (UniqueName: \"kubernetes.io/projected/88c40e7c-3418-457b-a367-0b24a45ea066-kube-api-access-5zrcf\") pod \"maas-controller-579cf7c6d-vlf2g\" (UID: \"88c40e7c-3418-457b-a367-0b24a45ea066\") " pod="opendatahub/maas-controller-579cf7c6d-vlf2g"
Oct 30 20:26:26.074454 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.074304    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c40e7c-3418-457b-a367-0b24a45ea066-metrics-certs\") pod \"maas-controller-579cf7c6d-vlf2g\" (UID: \"88c40e7c-3418-457b-a367-0b24a45ea066\") " pod="opendatahub/maas-controller-579cf7c6d-vlf2g"
Oct 30 20:26:26.074454 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.074344    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88c40e7c-3418-457b-a367-0b24a45ea066-webhook-cert\") pod \"maas-controller-579cf7c6d-vlf2g\" (UID: \"88c40e7c-3418-457b-a367-0b24a45ea066\") " pod="opendatahub/maas-controller-579cf7c6d-vlf2g"
Oct 30 20:26:26.079073 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.079057    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 20:26:26.079172 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.079156    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 20:26:26.079218 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.079172    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 20:26:26.079261 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.079248    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-42mmb\""
Oct 30 20:26:26.079406 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.079392    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 20:26:26.080478 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.080458    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-vlf2g"]
Oct 30 20:26:26.175375 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.175349    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zrcf\" (UniqueName: \"kubernetes.io/projected/88c40e7c-3418-457b-a367-0b24a45ea066-kube-api-access-5zrcf\") pod \"maas-controller-579cf7c6d-vlf2g\" (UID: \"88c40e7c-3418-457b-a367-0b24a45ea066\") " pod="opendatahub/maas-controller-579cf7c6d-vlf2g"
Oct 30 20:26:26.175500 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.175381    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c40e7c-3418-457b-a367-0b24a45ea066-metrics-certs\") pod \"maas-controller-579cf7c6d-vlf2g\" (UID: \"88c40e7c-3418-457b-a367-0b24a45ea066\") " pod="opendatahub/maas-controller-579cf7c6d-vlf2g"
Oct 30 20:26:26.175500 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.175406    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88c40e7c-3418-457b-a367-0b24a45ea066-webhook-cert\") pod \"maas-controller-579cf7c6d-vlf2g\" (UID: \"88c40e7c-3418-457b-a367-0b24a45ea066\") " pod="opendatahub/maas-controller-579cf7c6d-vlf2g"
Oct 30 20:26:26.177995 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.177968    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c40e7c-3418-457b-a367-0b24a45ea066-metrics-certs\") pod \"maas-controller-579cf7c6d-vlf2g\" (UID: \"88c40e7c-3418-457b-a367-0b24a45ea066\") " pod="opendatahub/maas-controller-579cf7c6d-vlf2g"
Oct 30 20:26:26.178108 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.178084    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88c40e7c-3418-457b-a367-0b24a45ea066-webhook-cert\") pod \"maas-controller-579cf7c6d-vlf2g\" (UID: \"88c40e7c-3418-457b-a367-0b24a45ea066\") " pod="opendatahub/maas-controller-579cf7c6d-vlf2g"
Oct 30 20:26:26.231747 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.231689    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zrcf\" (UniqueName: \"kubernetes.io/projected/88c40e7c-3418-457b-a367-0b24a45ea066-kube-api-access-5zrcf\") pod \"maas-controller-579cf7c6d-vlf2g\" (UID: \"88c40e7c-3418-457b-a367-0b24a45ea066\") " pod="opendatahub/maas-controller-579cf7c6d-vlf2g"
Oct 30 20:26:26.283728 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.283705    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-vlf2g"
Oct 30 20:26:26.384919 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.384891    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-756d9799fc-v9l8r"]
Oct 30 20:26:26.388175 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.388160    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-756d9799fc-v9l8r"
Oct 30 20:26:26.454047 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.454024    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-756d9799fc-v9l8r"]
Oct 30 20:26:26.478896 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.478875    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b-webhook-cert\") pod \"maas-controller-756d9799fc-v9l8r\" (UID: \"f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b\") " pod="opendatahub/maas-controller-756d9799fc-v9l8r"
Oct 30 20:26:26.479012 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.478935    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b-metrics-certs\") pod \"maas-controller-756d9799fc-v9l8r\" (UID: \"f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b\") " pod="opendatahub/maas-controller-756d9799fc-v9l8r"
Oct 30 20:26:26.479052 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.479024    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnw9j\" (UniqueName: \"kubernetes.io/projected/f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b-kube-api-access-tnw9j\") pod \"maas-controller-756d9799fc-v9l8r\" (UID: \"f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b\") " pod="opendatahub/maas-controller-756d9799fc-v9l8r"
Oct 30 20:26:26.579979 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.579909    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b-metrics-certs\") pod \"maas-controller-756d9799fc-v9l8r\" (UID: \"f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b\") " pod="opendatahub/maas-controller-756d9799fc-v9l8r"
Oct 30 20:26:26.579979 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.579951    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnw9j\" (UniqueName: \"kubernetes.io/projected/f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b-kube-api-access-tnw9j\") pod \"maas-controller-756d9799fc-v9l8r\" (UID: \"f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b\") " pod="opendatahub/maas-controller-756d9799fc-v9l8r"
Oct 30 20:26:26.580171 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.580150    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b-webhook-cert\") pod \"maas-controller-756d9799fc-v9l8r\" (UID: \"f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b\") " pod="opendatahub/maas-controller-756d9799fc-v9l8r"
Oct 30 20:26:26.582515 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.582494    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b-webhook-cert\") pod \"maas-controller-756d9799fc-v9l8r\" (UID: \"f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b\") " pod="opendatahub/maas-controller-756d9799fc-v9l8r"
Oct 30 20:26:26.582625 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.582516    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b-metrics-certs\") pod \"maas-controller-756d9799fc-v9l8r\" (UID: \"f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b\") " pod="opendatahub/maas-controller-756d9799fc-v9l8r"
Oct 30 20:26:26.636381 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.636363    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnw9j\" (UniqueName: \"kubernetes.io/projected/f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b-kube-api-access-tnw9j\") pod \"maas-controller-756d9799fc-v9l8r\" (UID: \"f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b\") " pod="opendatahub/maas-controller-756d9799fc-v9l8r"
Oct 30 20:26:26.649623 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.649605    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-vlf2g"]
Oct 30 20:26:26.652173 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:26:26.652155    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c40e7c_3418_457b_a367_0b24a45ea066.slice/crio-1abdd5551f7dc30544308cc751a5f990f43d941e2e940af676234a3bb796c05f WatchSource:0}: Error finding container 1abdd5551f7dc30544308cc751a5f990f43d941e2e940af676234a3bb796c05f: Status 404 returned error can't find the container with id 1abdd5551f7dc30544308cc751a5f990f43d941e2e940af676234a3bb796c05f
Oct 30 20:26:26.700426 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.700403    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-756d9799fc-v9l8r"
Oct 30 20:26:26.855066 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:26.854988    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-vlf2g" event={"ID":"88c40e7c-3418-457b-a367-0b24a45ea066","Type":"ContainerStarted","Data":"1abdd5551f7dc30544308cc751a5f990f43d941e2e940af676234a3bb796c05f"}
Oct 30 20:26:27.068700 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:27.068670    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-756d9799fc-v9l8r"]
Oct 30 20:26:27.071313 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:26:27.071286    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0cbbd23_0b3f_43e7_8b59_84d4cf81c14b.slice/crio-cebdba7aefcd5d15e15ee840bf28094a674e73bf718d51e8b370b291fe8aaa54 WatchSource:0}: Error finding container cebdba7aefcd5d15e15ee840bf28094a674e73bf718d51e8b370b291fe8aaa54: Status 404 returned error can't find the container with id cebdba7aefcd5d15e15ee840bf28094a674e73bf718d51e8b370b291fe8aaa54
Oct 30 20:26:27.860707 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:27.860658    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-756d9799fc-v9l8r" event={"ID":"f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b","Type":"ContainerStarted","Data":"cebdba7aefcd5d15e15ee840bf28094a674e73bf718d51e8b370b291fe8aaa54"}
Oct 30 20:26:30.955403 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:26:30.955361    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:26:32.885403 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:32.885359    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-vlf2g" event={"ID":"88c40e7c-3418-457b-a367-0b24a45ea066","Type":"ContainerStarted","Data":"b0cf71752ab6b00fb0cecca980dd88839ee018519f4d7df64aa8663fd23e5b73"}
Oct 30 20:26:32.885855 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:32.885434    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-vlf2g"
Oct 30 20:26:32.886749 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:32.886727    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-756d9799fc-v9l8r" event={"ID":"f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b","Type":"ContainerStarted","Data":"db10ae40390eefe009b5eee2236d8cf2dc3967cf43f26a8b9c602615abd35452"}
Oct 30 20:26:32.886876 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:32.886861    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-756d9799fc-v9l8r"
Oct 30 20:26:33.002474 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:33.002424    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-vlf2g" podStartSLOduration=2.830925746 podStartE2EDuration="8.002410456s" totalImagesPullingTime="5.17148471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:33.001221129 +0000 UTC m=+412.679849770" watchObservedRunningTime="2026-10-30 20:26:33.002410456 +0000 UTC m=+412.681039069"
Oct 30 20:26:33.119653 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:33.119596    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-756d9799fc-v9l8r" podStartSLOduration=2.198360683 podStartE2EDuration="7.119577671s" totalImagesPullingTime="4.921216988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:33.119348072 +0000 UTC m=+412.797976683" watchObservedRunningTime="2026-10-30 20:26:33.119577671 +0000 UTC m=+412.798206288"
Oct 30 20:26:35.166346 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:35.166289    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92"]
Oct 30 20:26:35.169440 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:35.169423    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92"
Oct 30 20:26:35.221087 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:35.221030    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 20:26:35.221437 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:35.221419    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 20:26:35.270639 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:35.270604    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 20:26:35.270868 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:35.270851    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-ggt8r\""
Oct 30 20:26:35.272854 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:35.272834    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92"]
Oct 30 20:26:35.353496 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:35.353464    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d131bdd1-ea77-4157-9304-8ddb194110b4-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5vs92\" (UID: \"d131bdd1-ea77-4157-9304-8ddb194110b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92"
Oct 30 20:26:35.353670 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:35.353517    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d131bdd1-ea77-4157-9304-8ddb194110b4-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5vs92\" (UID: \"d131bdd1-ea77-4157-9304-8ddb194110b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92"
Oct 30 20:26:35.353670 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:35.353615    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd7cx\" (UniqueName: \"kubernetes.io/projected/d131bdd1-ea77-4157-9304-8ddb194110b4-kube-api-access-hd7cx\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5vs92\" (UID: \"d131bdd1-ea77-4157-9304-8ddb194110b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92"
Oct 30 20:26:35.353670 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:35.353643    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d131bdd1-ea77-4157-9304-8ddb194110b4-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5vs92\" (UID: \"d131bdd1-ea77-4157-9304-8ddb194110b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92"
Oct 30 20:26:35.454332 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:35.454286    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd7cx\" (UniqueName: \"kubernetes.io/projected/d131bdd1-ea77-4157-9304-8ddb194110b4-kube-api-access-hd7cx\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5vs92\" (UID: \"d131bdd1-ea77-4157-9304-8ddb194110b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92"
Oct 30 20:26:35.454501 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:35.454341    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d131bdd1-ea77-4157-9304-8ddb194110b4-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5vs92\" (UID: \"d131bdd1-ea77-4157-9304-8ddb194110b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92"
Oct 30 20:26:35.454501 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:35.454367    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d131bdd1-ea77-4157-9304-8ddb194110b4-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5vs92\" (UID: \"d131bdd1-ea77-4157-9304-8ddb194110b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92"
Oct 30 20:26:35.454501 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:35.454397    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d131bdd1-ea77-4157-9304-8ddb194110b4-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5vs92\" (UID: \"d131bdd1-ea77-4157-9304-8ddb194110b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92"
Oct 30 20:26:35.454501 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:26:35.454439    4471 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Oct 30 20:26:35.454636 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:26:35.454517    4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d131bdd1-ea77-4157-9304-8ddb194110b4-proxy-source podName:d131bdd1-ea77-4157-9304-8ddb194110b4 nodeName:}" failed. No retries permitted until 2026-10-30 20:26:35.954500069 +0000 UTC m=+415.633128680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/d131bdd1-ea77-4157-9304-8ddb194110b4-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-5vs92" (UID: "d131bdd1-ea77-4157-9304-8ddb194110b4") : configmap "usage-tenancy-proxy-source" not found
Oct 30 20:26:35.455053 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:35.455029    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d131bdd1-ea77-4157-9304-8ddb194110b4-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5vs92\" (UID: \"d131bdd1-ea77-4157-9304-8ddb194110b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92"
Oct 30 20:26:35.456856 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:35.456827    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d131bdd1-ea77-4157-9304-8ddb194110b4-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5vs92\" (UID: \"d131bdd1-ea77-4157-9304-8ddb194110b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92"
Oct 30 20:26:35.513591 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:35.513531    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd7cx\" (UniqueName: \"kubernetes.io/projected/d131bdd1-ea77-4157-9304-8ddb194110b4-kube-api-access-hd7cx\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5vs92\" (UID: \"d131bdd1-ea77-4157-9304-8ddb194110b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92"
Oct 30 20:26:35.960008 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:35.959978    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d131bdd1-ea77-4157-9304-8ddb194110b4-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5vs92\" (UID: \"d131bdd1-ea77-4157-9304-8ddb194110b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92"
Oct 30 20:26:35.960763 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:35.960744    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/d131bdd1-ea77-4157-9304-8ddb194110b4-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5vs92\" (UID: \"d131bdd1-ea77-4157-9304-8ddb194110b4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92"
Oct 30 20:26:36.081411 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:36.081384    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92"
Oct 30 20:26:36.451435 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:36.451397    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92"]
Oct 30 20:26:36.453936 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:26:36.453911    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd131bdd1_ea77_4157_9304_8ddb194110b4.slice/crio-c25e95ed30c2fbbbd35b869c3255efc89ecea968a380649cc0ddbfca652847f4 WatchSource:0}: Error finding container c25e95ed30c2fbbbd35b869c3255efc89ecea968a380649cc0ddbfca652847f4: Status 404 returned error can't find the container with id c25e95ed30c2fbbbd35b869c3255efc89ecea968a380649cc0ddbfca652847f4
Oct 30 20:26:36.906190 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:36.906161    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92" event={"ID":"d131bdd1-ea77-4157-9304-8ddb194110b4","Type":"ContainerStarted","Data":"c25e95ed30c2fbbbd35b869c3255efc89ecea968a380649cc0ddbfca652847f4"}
Oct 30 20:26:43.892818 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:43.892788    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-vlf2g"
Oct 30 20:26:43.893740 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:43.893718    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-756d9799fc-v9l8r"
Oct 30 20:26:44.229270 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:44.229195    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-vlf2g"]
Oct 30 20:26:44.229490 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:44.229433    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-vlf2g" podUID="88c40e7c-3418-457b-a367-0b24a45ea066" containerName="manager" containerID="cri-o://b0cf71752ab6b00fb0cecca980dd88839ee018519f4d7df64aa8663fd23e5b73" gracePeriod=30
Oct 30 20:26:44.560683 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:44.560658    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-vlf2g"
Oct 30 20:26:44.630986 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:44.630959    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88c40e7c-3418-457b-a367-0b24a45ea066-webhook-cert\" (UniqueName: \"kubernetes.io/secret/88c40e7c-3418-457b-a367-0b24a45ea066-webhook-cert\") pod \"88c40e7c-3418-457b-a367-0b24a45ea066\" (UID: \"88c40e7c-3418-457b-a367-0b24a45ea066\") "
Oct 30 20:26:44.631152 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:44.631002    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88c40e7c-3418-457b-a367-0b24a45ea066-metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c40e7c-3418-457b-a367-0b24a45ea066-metrics-certs\") pod \"88c40e7c-3418-457b-a367-0b24a45ea066\" (UID: \"88c40e7c-3418-457b-a367-0b24a45ea066\") "
Oct 30 20:26:44.631152 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:44.631064    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88c40e7c-3418-457b-a367-0b24a45ea066-kube-api-access-5zrcf\" (UniqueName: \"kubernetes.io/projected/88c40e7c-3418-457b-a367-0b24a45ea066-kube-api-access-5zrcf\") pod \"88c40e7c-3418-457b-a367-0b24a45ea066\" (UID: \"88c40e7c-3418-457b-a367-0b24a45ea066\") "
Oct 30 20:26:44.633241 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:44.633210    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88c40e7c-3418-457b-a367-0b24a45ea066-metrics-certs" pod "88c40e7c-3418-457b-a367-0b24a45ea066" (UID: "88c40e7c-3418-457b-a367-0b24a45ea066"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:26:44.633355 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:44.633243    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88c40e7c-3418-457b-a367-0b24a45ea066-webhook-cert" pod "88c40e7c-3418-457b-a367-0b24a45ea066" (UID: "88c40e7c-3418-457b-a367-0b24a45ea066"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:26:44.633355 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:44.633310    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c40e7c-3418-457b-a367-0b24a45ea066-kube-api-access-5zrcf" pod "88c40e7c-3418-457b-a367-0b24a45ea066" (UID: "88c40e7c-3418-457b-a367-0b24a45ea066"). InnerVolumeSpecName "kube-api-access-5zrcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:44.731764 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:44.731718    4471 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88c40e7c-3418-457b-a367-0b24a45ea066-webhook-cert\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:44.731764 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:44.731755    4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88c40e7c-3418-457b-a367-0b24a45ea066-metrics-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:44.731957 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:44.731771    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zrcf\" (UniqueName: \"kubernetes.io/projected/88c40e7c-3418-457b-a367-0b24a45ea066-kube-api-access-5zrcf\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:44.941792 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:44.941753    4471 generic.go:356] "Generic (PLEG): container finished" podID="88c40e7c-3418-457b-a367-0b24a45ea066" containerID="b0cf71752ab6b00fb0cecca980dd88839ee018519f4d7df64aa8663fd23e5b73" exitCode=0
Oct 30 20:26:44.941792 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:44.941792    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-vlf2g"
Oct 30 20:26:44.942721 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:44.941843    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-vlf2g" event={"ID":"88c40e7c-3418-457b-a367-0b24a45ea066","Type":"ContainerDied","Data":"b0cf71752ab6b00fb0cecca980dd88839ee018519f4d7df64aa8663fd23e5b73"}
Oct 30 20:26:44.942721 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:44.941893    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-vlf2g" event={"ID":"88c40e7c-3418-457b-a367-0b24a45ea066","Type":"ContainerDied","Data":"1abdd5551f7dc30544308cc751a5f990f43d941e2e940af676234a3bb796c05f"}
Oct 30 20:26:44.942721 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:44.941921    4471 scope.go:122] "RemoveContainer" containerID="b0cf71752ab6b00fb0cecca980dd88839ee018519f4d7df64aa8663fd23e5b73"
Oct 30 20:26:44.950726 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:44.950700    4471 scope.go:122] "RemoveContainer" containerID="b0cf71752ab6b00fb0cecca980dd88839ee018519f4d7df64aa8663fd23e5b73"
Oct 30 20:26:44.951103 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:26:44.951083    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0cf71752ab6b00fb0cecca980dd88839ee018519f4d7df64aa8663fd23e5b73\": container with ID starting with b0cf71752ab6b00fb0cecca980dd88839ee018519f4d7df64aa8663fd23e5b73 not found: ID does not exist" containerID="b0cf71752ab6b00fb0cecca980dd88839ee018519f4d7df64aa8663fd23e5b73"
Oct 30 20:26:44.951203 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:44.951110    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0cf71752ab6b00fb0cecca980dd88839ee018519f4d7df64aa8663fd23e5b73"} err="failed to get container status \"b0cf71752ab6b00fb0cecca980dd88839ee018519f4d7df64aa8663fd23e5b73\": rpc error: code = NotFound desc = could not find container \"b0cf71752ab6b00fb0cecca980dd88839ee018519f4d7df64aa8663fd23e5b73\": container with ID starting with b0cf71752ab6b00fb0cecca980dd88839ee018519f4d7df64aa8663fd23e5b73 not found: ID does not exist"
Oct 30 20:26:45.106127 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:45.106093    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-vlf2g"]
Oct 30 20:26:45.121177 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:45.121148    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-vlf2g"]
Oct 30 20:26:46.433967 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:46.433928    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88c40e7c-3418-457b-a367-0b24a45ea066" path="/var/lib/kubelet/pods/88c40e7c-3418-457b-a367-0b24a45ea066/volumes"
Oct 30 20:26:49.966177 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:49.966138    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92" event={"ID":"d131bdd1-ea77-4157-9304-8ddb194110b4","Type":"ContainerStarted","Data":"c69b3f6e37e1f7d0a5b99f4ed0cd30293d707cda97b17ee4f4bc1419c535d4ce"}
Oct 30 20:26:49.966556 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:49.966380    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92"
Oct 30 20:26:50.086659 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:50.086605    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92" podStartSLOduration=1.704334233 podStartE2EDuration="15.086590749s" totalImagesPullingTime="13.382256516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:50.085071771 +0000 UTC m=+429.763700386" watchObservedRunningTime="2026-10-30 20:26:50.086590749 +0000 UTC m=+429.765219363"
Oct 30 20:26:51.244515 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.244481    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"]
Oct 30 20:26:51.244906 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.244751    4471 cpu_manager.go:414] "containerMap: removing container" podUID="88c40e7c-3418-457b-a367-0b24a45ea066" containerName="manager"
Oct 30 20:26:51.244906 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.244760    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88c40e7c-3418-457b-a367-0b24a45ea066" containerName="manager"
Oct 30 20:26:51.244906 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.244817    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="88c40e7c-3418-457b-a367-0b24a45ea066" containerName="manager"
Oct 30 20:26:51.247730 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.247714    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.289547 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.289521    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cf792592-4698-4010-bfb6-d6efef407058-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rshc\" (UID: \"cf792592-4698-4010-bfb6-d6efef407058\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.289676 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.289552    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wfqt\" (UniqueName: \"kubernetes.io/projected/cf792592-4698-4010-bfb6-d6efef407058-kube-api-access-2wfqt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rshc\" (UID: \"cf792592-4698-4010-bfb6-d6efef407058\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.289676 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.289573    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf792592-4698-4010-bfb6-d6efef407058-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rshc\" (UID: \"cf792592-4698-4010-bfb6-d6efef407058\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.289676 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.289632    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cf792592-4698-4010-bfb6-d6efef407058-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rshc\" (UID: \"cf792592-4698-4010-bfb6-d6efef407058\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.289676 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.289661    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cf792592-4698-4010-bfb6-d6efef407058-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rshc\" (UID: \"cf792592-4698-4010-bfb6-d6efef407058\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.289820 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.289768    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cf792592-4698-4010-bfb6-d6efef407058-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rshc\" (UID: \"cf792592-4698-4010-bfb6-d6efef407058\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.289820 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.289808    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf792592-4698-4010-bfb6-d6efef407058-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rshc\" (UID: \"cf792592-4698-4010-bfb6-d6efef407058\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.350966 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.350936    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"]
Oct 30 20:26:51.390382 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.390354    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cf792592-4698-4010-bfb6-d6efef407058-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rshc\" (UID: \"cf792592-4698-4010-bfb6-d6efef407058\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.390533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.390389    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wfqt\" (UniqueName: \"kubernetes.io/projected/cf792592-4698-4010-bfb6-d6efef407058-kube-api-access-2wfqt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rshc\" (UID: \"cf792592-4698-4010-bfb6-d6efef407058\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.390533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.390406    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf792592-4698-4010-bfb6-d6efef407058-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rshc\" (UID: \"cf792592-4698-4010-bfb6-d6efef407058\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.390533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.390429    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cf792592-4698-4010-bfb6-d6efef407058-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rshc\" (UID: \"cf792592-4698-4010-bfb6-d6efef407058\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.390533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.390456    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cf792592-4698-4010-bfb6-d6efef407058-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rshc\" (UID: \"cf792592-4698-4010-bfb6-d6efef407058\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.390533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.390482    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cf792592-4698-4010-bfb6-d6efef407058-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rshc\" (UID: \"cf792592-4698-4010-bfb6-d6efef407058\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.390533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.390517    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf792592-4698-4010-bfb6-d6efef407058-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rshc\" (UID: \"cf792592-4698-4010-bfb6-d6efef407058\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.391460 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.391429    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cf792592-4698-4010-bfb6-d6efef407058-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rshc\" (UID: \"cf792592-4698-4010-bfb6-d6efef407058\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.392904 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.392877    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cf792592-4698-4010-bfb6-d6efef407058-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rshc\" (UID: \"cf792592-4698-4010-bfb6-d6efef407058\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.392994 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.392902    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf792592-4698-4010-bfb6-d6efef407058-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rshc\" (UID: \"cf792592-4698-4010-bfb6-d6efef407058\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.392994 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.392966    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cf792592-4698-4010-bfb6-d6efef407058-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rshc\" (UID: \"cf792592-4698-4010-bfb6-d6efef407058\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.393059 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.393008    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cf792592-4698-4010-bfb6-d6efef407058-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rshc\" (UID: \"cf792592-4698-4010-bfb6-d6efef407058\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.447420 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.447394    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf792592-4698-4010-bfb6-d6efef407058-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rshc\" (UID: \"cf792592-4698-4010-bfb6-d6efef407058\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.447615 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.447588    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wfqt\" (UniqueName: \"kubernetes.io/projected/cf792592-4698-4010-bfb6-d6efef407058-kube-api-access-2wfqt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rshc\" (UID: \"cf792592-4698-4010-bfb6-d6efef407058\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.558879 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.558823    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:51.984038 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.984016    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"]
Oct 30 20:26:51.986439 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:26:51.986406    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf792592_4698_4010_bfb6_d6efef407058.slice/crio-21b8d89831fac76eaca8761c889397f230315896c3915978834d7b8f0629fe38 WatchSource:0}: Error finding container 21b8d89831fac76eaca8761c889397f230315896c3915978834d7b8f0629fe38: Status 404 returned error can't find the container with id 21b8d89831fac76eaca8761c889397f230315896c3915978834d7b8f0629fe38
Oct 30 20:26:51.988390 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.988358    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:26:51.988490 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:51.988419    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:26:52.979461 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:52.979424    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc" event={"ID":"cf792592-4698-4010-bfb6-d6efef407058","Type":"ContainerStarted","Data":"3e1320371340ddbca3c7b90dab45963fd9562d7fd5d025b2dd0bdddcfa0b6239"}
Oct 30 20:26:52.979925 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:52.979471    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc" event={"ID":"cf792592-4698-4010-bfb6-d6efef407058","Type":"ContainerStarted","Data":"21b8d89831fac76eaca8761c889397f230315896c3915978834d7b8f0629fe38"}
Oct 30 20:26:52.979925 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:52.979665    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:53.198572 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:53.198523    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc" podStartSLOduration=2.198508113 podStartE2EDuration="2.198508113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:53.148637658 +0000 UTC m=+432.827266273" watchObservedRunningTime="2026-10-30 20:26:53.198508113 +0000 UTC m=+432.877136727"
Oct 30 20:26:53.984878 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:53.984848    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rshc"
Oct 30 20:26:54.202855 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.202826    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"]
Oct 30 20:26:54.203034 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.203015    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888" podUID="6357c4ef-e469-4a75-b456-71167a77d985" containerName="discovery" containerID="cri-o://290f24b4c175d29e95432dea6e62b2bc4bf6734f05756df458188491d7afab04" gracePeriod=30
Oct 30 20:26:54.442089 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.442065    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:26:54.515673 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.515646    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6357c4ef-e469-4a75-b456-71167a77d985-istio-token\" (UniqueName: \"kubernetes.io/projected/6357c4ef-e469-4a75-b456-71167a77d985-istio-token\") pod \"6357c4ef-e469-4a75-b456-71167a77d985\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") "
Oct 30 20:26:54.515828 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.515690    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6357c4ef-e469-4a75-b456-71167a77d985-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6357c4ef-e469-4a75-b456-71167a77d985-local-certs\") pod \"6357c4ef-e469-4a75-b456-71167a77d985\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") "
Oct 30 20:26:54.515828 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.515711    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6357c4ef-e469-4a75-b456-71167a77d985-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6357c4ef-e469-4a75-b456-71167a77d985-istio-kubeconfig\") pod \"6357c4ef-e469-4a75-b456-71167a77d985\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") "
Oct 30 20:26:54.515828 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.515764    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6357c4ef-e469-4a75-b456-71167a77d985-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6357c4ef-e469-4a75-b456-71167a77d985-istio-csr-ca-configmap\") pod \"6357c4ef-e469-4a75-b456-71167a77d985\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") "
Oct 30 20:26:54.515828 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.515784    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6357c4ef-e469-4a75-b456-71167a77d985-cacerts\" (UniqueName: \"kubernetes.io/secret/6357c4ef-e469-4a75-b456-71167a77d985-cacerts\") pod \"6357c4ef-e469-4a75-b456-71167a77d985\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") "
Oct 30 20:26:54.515828 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.515806    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6357c4ef-e469-4a75-b456-71167a77d985-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6357c4ef-e469-4a75-b456-71167a77d985-istio-csr-dns-cert\") pod \"6357c4ef-e469-4a75-b456-71167a77d985\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") "
Oct 30 20:26:54.516074 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.515843    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6357c4ef-e469-4a75-b456-71167a77d985-kube-api-access-cptkt\" (UniqueName: \"kubernetes.io/projected/6357c4ef-e469-4a75-b456-71167a77d985-kube-api-access-cptkt\") pod \"6357c4ef-e469-4a75-b456-71167a77d985\" (UID: \"6357c4ef-e469-4a75-b456-71167a77d985\") "
Oct 30 20:26:54.516478 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.516436    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6357c4ef-e469-4a75-b456-71167a77d985-istio-csr-ca-configmap" pod "6357c4ef-e469-4a75-b456-71167a77d985" (UID: "6357c4ef-e469-4a75-b456-71167a77d985"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:54.518133 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.518104    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6357c4ef-e469-4a75-b456-71167a77d985-istio-kubeconfig" pod "6357c4ef-e469-4a75-b456-71167a77d985" (UID: "6357c4ef-e469-4a75-b456-71167a77d985"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:26:54.518305 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.518282    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6357c4ef-e469-4a75-b456-71167a77d985-kube-api-access-cptkt" pod "6357c4ef-e469-4a75-b456-71167a77d985" (UID: "6357c4ef-e469-4a75-b456-71167a77d985"). InnerVolumeSpecName "kube-api-access-cptkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:54.518442 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.518419    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6357c4ef-e469-4a75-b456-71167a77d985-local-certs" pod "6357c4ef-e469-4a75-b456-71167a77d985" (UID: "6357c4ef-e469-4a75-b456-71167a77d985"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:54.518507 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.518465    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6357c4ef-e469-4a75-b456-71167a77d985-cacerts" pod "6357c4ef-e469-4a75-b456-71167a77d985" (UID: "6357c4ef-e469-4a75-b456-71167a77d985"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:26:54.518561 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.518508    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6357c4ef-e469-4a75-b456-71167a77d985-istio-token" pod "6357c4ef-e469-4a75-b456-71167a77d985" (UID: "6357c4ef-e469-4a75-b456-71167a77d985"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:54.518684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.518663    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6357c4ef-e469-4a75-b456-71167a77d985-istio-csr-dns-cert" pod "6357c4ef-e469-4a75-b456-71167a77d985" (UID: "6357c4ef-e469-4a75-b456-71167a77d985"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:26:54.617376 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.617295    4471 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6357c4ef-e469-4a75-b456-71167a77d985-istio-csr-ca-configmap\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.617376 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.617344    4471 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6357c4ef-e469-4a75-b456-71167a77d985-cacerts\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.617376 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.617359    4471 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6357c4ef-e469-4a75-b456-71167a77d985-istio-csr-dns-cert\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.617376 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.617374    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cptkt\" (UniqueName: \"kubernetes.io/projected/6357c4ef-e469-4a75-b456-71167a77d985-kube-api-access-cptkt\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.617588 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.617388    4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6357c4ef-e469-4a75-b456-71167a77d985-istio-token\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.617588 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.617400    4471 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6357c4ef-e469-4a75-b456-71167a77d985-local-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.617588 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.617412    4471 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6357c4ef-e469-4a75-b456-71167a77d985-istio-kubeconfig\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:54.988419 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.988379    4471 generic.go:356] "Generic (PLEG): container finished" podID="6357c4ef-e469-4a75-b456-71167a77d985" containerID="290f24b4c175d29e95432dea6e62b2bc4bf6734f05756df458188491d7afab04" exitCode=0
Oct 30 20:26:54.988885 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.988427    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"
Oct 30 20:26:54.988885 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.988470    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888" event={"ID":"6357c4ef-e469-4a75-b456-71167a77d985","Type":"ContainerDied","Data":"290f24b4c175d29e95432dea6e62b2bc4bf6734f05756df458188491d7afab04"}
Oct 30 20:26:54.988885 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.988528    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888" event={"ID":"6357c4ef-e469-4a75-b456-71167a77d985","Type":"ContainerDied","Data":"413b6026f6a3996cca15c72242fa303e24c6da486b27b4e9570125638f88ac91"}
Oct 30 20:26:54.988885 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.988556    4471 scope.go:122] "RemoveContainer" containerID="290f24b4c175d29e95432dea6e62b2bc4bf6734f05756df458188491d7afab04"
Oct 30 20:26:54.998088 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.998071    4471 scope.go:122] "RemoveContainer" containerID="290f24b4c175d29e95432dea6e62b2bc4bf6734f05756df458188491d7afab04"
Oct 30 20:26:54.998386 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:26:54.998364    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"290f24b4c175d29e95432dea6e62b2bc4bf6734f05756df458188491d7afab04\": container with ID starting with 290f24b4c175d29e95432dea6e62b2bc4bf6734f05756df458188491d7afab04 not found: ID does not exist" containerID="290f24b4c175d29e95432dea6e62b2bc4bf6734f05756df458188491d7afab04"
Oct 30 20:26:54.998485 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:54.998392    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"290f24b4c175d29e95432dea6e62b2bc4bf6734f05756df458188491d7afab04"} err="failed to get container status \"290f24b4c175d29e95432dea6e62b2bc4bf6734f05756df458188491d7afab04\": rpc error: code = NotFound desc = could not find container \"290f24b4c175d29e95432dea6e62b2bc4bf6734f05756df458188491d7afab04\": container with ID starting with 290f24b4c175d29e95432dea6e62b2bc4bf6734f05756df458188491d7afab04 not found: ID does not exist"
Oct 30 20:26:55.153502 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:55.153470    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"]
Oct 30 20:26:55.202827 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:55.202799    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888"]
Oct 30 20:26:56.428684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:56.428644    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6357c4ef-e469-4a75-b456-71167a77d985" path="/var/lib/kubelet/pods/6357c4ef-e469-4a75-b456-71167a77d985/volumes"
Oct 30 20:26:56.765845 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:56.765779    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xbwkj"]
Oct 30 20:26:56.766063 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:56.766051    4471 cpu_manager.go:414] "containerMap: removing container" podUID="6357c4ef-e469-4a75-b456-71167a77d985" containerName="discovery"
Oct 30 20:26:56.766110 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:56.766064    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6357c4ef-e469-4a75-b456-71167a77d985" containerName="discovery"
Oct 30 20:26:56.766142 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:56.766121    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="6357c4ef-e469-4a75-b456-71167a77d985" containerName="discovery"
Oct 30 20:26:56.768609 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:56.768594    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-xbwkj"
Oct 30 20:26:56.830487 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:56.830464    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 20:26:56.831522 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:56.831504    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-w4lq2\""
Oct 30 20:26:56.836113 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:56.836081    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsp9p\" (UniqueName: \"kubernetes.io/projected/a7f450f5-34f5-47a4-96ed-9fb59b9544fa-kube-api-access-bsp9p\") pod \"kserve-controller-manager-776cc66d47-xbwkj\" (UID: \"a7f450f5-34f5-47a4-96ed-9fb59b9544fa\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xbwkj"
Oct 30 20:26:56.836216 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:56.836128    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7f450f5-34f5-47a4-96ed-9fb59b9544fa-cert\") pod \"kserve-controller-manager-776cc66d47-xbwkj\" (UID: \"a7f450f5-34f5-47a4-96ed-9fb59b9544fa\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xbwkj"
Oct 30 20:26:56.837208 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:56.837187    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xbwkj"]
Oct 30 20:26:56.915297 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:56.915272    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69"]
Oct 30 20:26:56.917157 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:56.917143    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69"
Oct 30 20:26:56.937569 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:56.937549    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsp9p\" (UniqueName: \"kubernetes.io/projected/a7f450f5-34f5-47a4-96ed-9fb59b9544fa-kube-api-access-bsp9p\") pod \"kserve-controller-manager-776cc66d47-xbwkj\" (UID: \"a7f450f5-34f5-47a4-96ed-9fb59b9544fa\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xbwkj"
Oct 30 20:26:56.937687 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:56.937600    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7f450f5-34f5-47a4-96ed-9fb59b9544fa-cert\") pod \"kserve-controller-manager-776cc66d47-xbwkj\" (UID: \"a7f450f5-34f5-47a4-96ed-9fb59b9544fa\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xbwkj"
Oct 30 20:26:56.940162 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:56.940143    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7f450f5-34f5-47a4-96ed-9fb59b9544fa-cert\") pod \"kserve-controller-manager-776cc66d47-xbwkj\" (UID: \"a7f450f5-34f5-47a4-96ed-9fb59b9544fa\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xbwkj"
Oct 30 20:26:56.969627 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:56.969607    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-9g8v6\""
Oct 30 20:26:56.969695 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:56.969637    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 20:26:56.978991 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:56.978966    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69"]
Oct 30 20:26:57.004209 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:57.004183    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsp9p\" (UniqueName: \"kubernetes.io/projected/a7f450f5-34f5-47a4-96ed-9fb59b9544fa-kube-api-access-bsp9p\") pod \"kserve-controller-manager-776cc66d47-xbwkj\" (UID: \"a7f450f5-34f5-47a4-96ed-9fb59b9544fa\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xbwkj"
Oct 30 20:26:57.038962 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:57.038898    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhgz6\" (UniqueName: \"kubernetes.io/projected/2b8c9d4e-47d2-404c-b13f-ddfcd695ff15-kube-api-access-lhgz6\") pod \"llmisvc-controller-manager-869d5dcc5d-p4w69\" (UID: \"2b8c9d4e-47d2-404c-b13f-ddfcd695ff15\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69"
Oct 30 20:26:57.038962 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:57.038945    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b8c9d4e-47d2-404c-b13f-ddfcd695ff15-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-p4w69\" (UID: \"2b8c9d4e-47d2-404c-b13f-ddfcd695ff15\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69"
Oct 30 20:26:57.079362 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:57.079335    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-xbwkj"
Oct 30 20:26:57.139876 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:57.139846    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhgz6\" (UniqueName: \"kubernetes.io/projected/2b8c9d4e-47d2-404c-b13f-ddfcd695ff15-kube-api-access-lhgz6\") pod \"llmisvc-controller-manager-869d5dcc5d-p4w69\" (UID: \"2b8c9d4e-47d2-404c-b13f-ddfcd695ff15\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69"
Oct 30 20:26:57.139991 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:57.139906    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b8c9d4e-47d2-404c-b13f-ddfcd695ff15-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-p4w69\" (UID: \"2b8c9d4e-47d2-404c-b13f-ddfcd695ff15\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69"
Oct 30 20:26:57.142386 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:57.142362    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b8c9d4e-47d2-404c-b13f-ddfcd695ff15-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-p4w69\" (UID: \"2b8c9d4e-47d2-404c-b13f-ddfcd695ff15\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69"
Oct 30 20:26:57.197142 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:57.197105    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhgz6\" (UniqueName: \"kubernetes.io/projected/2b8c9d4e-47d2-404c-b13f-ddfcd695ff15-kube-api-access-lhgz6\") pod \"llmisvc-controller-manager-869d5dcc5d-p4w69\" (UID: \"2b8c9d4e-47d2-404c-b13f-ddfcd695ff15\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69"
Oct 30 20:26:57.227109 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:57.227082    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69"
Oct 30 20:26:57.445359 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:57.445308    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xbwkj"]
Oct 30 20:26:57.447907 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:26:57.447882    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f450f5_34f5_47a4_96ed_9fb59b9544fa.slice/crio-d2b4a7031bf37d21be56d11a537b1340617dd293eadf6ed9c2518cffe1a913f3 WatchSource:0}: Error finding container d2b4a7031bf37d21be56d11a537b1340617dd293eadf6ed9c2518cffe1a913f3: Status 404 returned error can't find the container with id d2b4a7031bf37d21be56d11a537b1340617dd293eadf6ed9c2518cffe1a913f3
Oct 30 20:26:57.590957 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:57.590932    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69"]
Oct 30 20:26:57.593367 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:26:57.593340    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2b8c9d4e_47d2_404c_b13f_ddfcd695ff15.slice/crio-02bf59907633a46dd2da6c8167b0daa88cc7f6a530150159f91d40dbb7303bea WatchSource:0}: Error finding container 02bf59907633a46dd2da6c8167b0daa88cc7f6a530150159f91d40dbb7303bea: Status 404 returned error can't find the container with id 02bf59907633a46dd2da6c8167b0daa88cc7f6a530150159f91d40dbb7303bea
Oct 30 20:26:58.004211 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:58.004174    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-xbwkj" event={"ID":"a7f450f5-34f5-47a4-96ed-9fb59b9544fa","Type":"ContainerStarted","Data":"d2b4a7031bf37d21be56d11a537b1340617dd293eadf6ed9c2518cffe1a913f3"}
Oct 30 20:26:58.005197 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:58.005175    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69" event={"ID":"2b8c9d4e-47d2-404c-b13f-ddfcd695ff15","Type":"ContainerStarted","Data":"02bf59907633a46dd2da6c8167b0daa88cc7f6a530150159f91d40dbb7303bea"}
Oct 30 20:26:59.382020 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:59.381980    4471 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-lc888 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.17:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:26:59.382473 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:26:59.382044    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lc888" podUID="6357c4ef-e469-4a75-b456-71167a77d985" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.17:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:27:00.017837 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:00.017763    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-xbwkj" event={"ID":"a7f450f5-34f5-47a4-96ed-9fb59b9544fa","Type":"ContainerStarted","Data":"4f2a536b99bec3ca316961a5d4657aa3b92ff0994eab0e7d0b754a0a79c4d664"}
Oct 30 20:27:00.017985 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:00.017860    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-xbwkj"
Oct 30 20:27:00.133886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:00.133830    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-xbwkj" podStartSLOduration=1.835951544 podStartE2EDuration="4.133814905s" totalImagesPullingTime="2.297863361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:00.132369809 +0000 UTC m=+439.810998425" watchObservedRunningTime="2026-10-30 20:27:00.133814905 +0000 UTC m=+439.812443519"
Oct 30 20:27:00.954429 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:27:00.954304    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:27:00.975502 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:00.975475    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5vs92"
Oct 30 20:27:01.023104 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:01.023074    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69" event={"ID":"2b8c9d4e-47d2-404c-b13f-ddfcd695ff15","Type":"ContainerStarted","Data":"c90d7dc570ab8725447d5a630341a43ba7b0c97d1505932a81e551eb4254e46c"}
Oct 30 20:27:01.217545 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:01.217494    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69" podStartSLOduration=1.947174161 podStartE2EDuration="5.217474397s" totalImagesPullingTime="3.270300236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:01.215797558 +0000 UTC m=+440.894426172" watchObservedRunningTime="2026-10-30 20:27:01.217474397 +0000 UTC m=+440.896103013"
Oct 30 20:27:02.028111 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:02.028084    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69"
Oct 30 20:27:07.052008 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:07.051977    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"]
Oct 30 20:27:07.056397 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:07.056378    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"
Oct 30 20:27:07.114616 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:07.114593    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pl8x\" (UniqueName: \"kubernetes.io/projected/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-kube-api-access-8pl8x\") pod \"maas-api-6d669c7fcd-5ndqs\" (UID: \"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"
Oct 30 20:27:07.114754 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:07.114630    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-openshift-service-ca\") pod \"maas-api-6d669c7fcd-5ndqs\" (UID: \"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"
Oct 30 20:27:07.114754 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:07.114714    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-maas-api-tls\") pod \"maas-api-6d669c7fcd-5ndqs\" (UID: \"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"
Oct 30 20:27:07.114831 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:07.114759    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-metrics-certs\") pod \"maas-api-6d669c7fcd-5ndqs\" (UID: \"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"
Oct 30 20:27:07.157643 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:07.157617    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 20:27:07.157643 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:07.157621    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xgxz5\""
Oct 30 20:27:07.157813 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:07.157759    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 20:27:07.159046 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:07.159027    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"]
Oct 30 20:27:07.215099 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:07.215076    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-metrics-certs\") pod \"maas-api-6d669c7fcd-5ndqs\" (UID: \"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"
Oct 30 20:27:07.215269 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:07.215107    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pl8x\" (UniqueName: \"kubernetes.io/projected/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-kube-api-access-8pl8x\") pod \"maas-api-6d669c7fcd-5ndqs\" (UID: \"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"
Oct 30 20:27:07.215269 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:07.215136    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-openshift-service-ca\") pod \"maas-api-6d669c7fcd-5ndqs\" (UID: \"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"
Oct 30 20:27:07.215269 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:07.215207    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-maas-api-tls\") pod \"maas-api-6d669c7fcd-5ndqs\" (UID: \"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"
Oct 30 20:27:07.215805 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:07.215786    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-openshift-service-ca\") pod \"maas-api-6d669c7fcd-5ndqs\" (UID: \"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"
Oct 30 20:27:07.217577 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:07.217551    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-metrics-certs\") pod \"maas-api-6d669c7fcd-5ndqs\" (UID: \"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"
Oct 30 20:27:07.217672 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:07.217561    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-maas-api-tls\") pod \"maas-api-6d669c7fcd-5ndqs\" (UID: \"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"
Oct 30 20:27:07.271819 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:07.271792    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pl8x\" (UniqueName: \"kubernetes.io/projected/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-kube-api-access-8pl8x\") pod \"maas-api-6d669c7fcd-5ndqs\" (UID: \"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"
Oct 30 20:27:07.367567 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:07.367506    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"
Oct 30 20:27:07.738549 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:07.738515    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"]
Oct 30 20:27:07.741122 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:27:07.741096    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4985eb6f_fd0a_4dc9_951a_0ec5e6ea9d82.slice/crio-ac83fba8aee23783b8001292714d8c6e98c717f32fdaeb060f7cb66861ff860c WatchSource:0}: Error finding container ac83fba8aee23783b8001292714d8c6e98c717f32fdaeb060f7cb66861ff860c: Status 404 returned error can't find the container with id ac83fba8aee23783b8001292714d8c6e98c717f32fdaeb060f7cb66861ff860c
Oct 30 20:27:08.053157 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:08.053083    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs" event={"ID":"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82","Type":"ContainerStarted","Data":"ac83fba8aee23783b8001292714d8c6e98c717f32fdaeb060f7cb66861ff860c"}
Oct 30 20:27:10.062424 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:10.062313    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs" event={"ID":"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82","Type":"ContainerStarted","Data":"d44afcc5f4cf6cfd8d8c9345ede764827eee1473efe205c3a8e4a1ad4be89de8"}
Oct 30 20:27:10.062424 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:10.062414    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"
Oct 30 20:27:10.178789 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:10.178728    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs" podStartSLOduration=1.195377832 podStartE2EDuration="3.178712292s" totalImagesPullingTime="1.98333446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:10.178559191 +0000 UTC m=+449.857187806" watchObservedRunningTime="2026-10-30 20:27:10.178712292 +0000 UTC m=+449.857340911"
Oct 30 20:27:16.071960 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:16.071931    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"
Oct 30 20:27:17.155623 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:17.155591    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"]
Oct 30 20:27:17.160508 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:17.160486    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"
Oct 30 20:27:17.267837 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:17.267801    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"]
Oct 30 20:27:17.294964 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:17.294937    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk9r2\" (UniqueName: \"kubernetes.io/projected/07769876-3c0c-4714-b6c7-bbd195c0d0c6-kube-api-access-tk9r2\") pod \"maas-api-94dffd7db-65cxx\" (UID: \"07769876-3c0c-4714-b6c7-bbd195c0d0c6\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"
Oct 30 20:27:17.295120 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:17.294976    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07769876-3c0c-4714-b6c7-bbd195c0d0c6-metrics-certs\") pod \"maas-api-94dffd7db-65cxx\" (UID: \"07769876-3c0c-4714-b6c7-bbd195c0d0c6\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"
Oct 30 20:27:17.295120 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:17.295079    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07769876-3c0c-4714-b6c7-bbd195c0d0c6-openshift-service-ca\") pod \"maas-api-94dffd7db-65cxx\" (UID: \"07769876-3c0c-4714-b6c7-bbd195c0d0c6\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"
Oct 30 20:27:17.295120 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:17.295101    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07769876-3c0c-4714-b6c7-bbd195c0d0c6-maas-api-tls\") pod \"maas-api-94dffd7db-65cxx\" (UID: \"07769876-3c0c-4714-b6c7-bbd195c0d0c6\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"
Oct 30 20:27:17.395901 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:17.395874    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07769876-3c0c-4714-b6c7-bbd195c0d0c6-openshift-service-ca\") pod \"maas-api-94dffd7db-65cxx\" (UID: \"07769876-3c0c-4714-b6c7-bbd195c0d0c6\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"
Oct 30 20:27:17.396038 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:17.395906    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07769876-3c0c-4714-b6c7-bbd195c0d0c6-maas-api-tls\") pod \"maas-api-94dffd7db-65cxx\" (UID: \"07769876-3c0c-4714-b6c7-bbd195c0d0c6\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"
Oct 30 20:27:17.396038 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:17.395935    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9r2\" (UniqueName: \"kubernetes.io/projected/07769876-3c0c-4714-b6c7-bbd195c0d0c6-kube-api-access-tk9r2\") pod \"maas-api-94dffd7db-65cxx\" (UID: \"07769876-3c0c-4714-b6c7-bbd195c0d0c6\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"
Oct 30 20:27:17.396038 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:17.396014    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07769876-3c0c-4714-b6c7-bbd195c0d0c6-metrics-certs\") pod \"maas-api-94dffd7db-65cxx\" (UID: \"07769876-3c0c-4714-b6c7-bbd195c0d0c6\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"
Oct 30 20:27:17.396633 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:17.396611    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07769876-3c0c-4714-b6c7-bbd195c0d0c6-openshift-service-ca\") pod \"maas-api-94dffd7db-65cxx\" (UID: \"07769876-3c0c-4714-b6c7-bbd195c0d0c6\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"
Oct 30 20:27:17.398580 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:17.398561    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07769876-3c0c-4714-b6c7-bbd195c0d0c6-metrics-certs\") pod \"maas-api-94dffd7db-65cxx\" (UID: \"07769876-3c0c-4714-b6c7-bbd195c0d0c6\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"
Oct 30 20:27:17.398668 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:17.398591    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07769876-3c0c-4714-b6c7-bbd195c0d0c6-maas-api-tls\") pod \"maas-api-94dffd7db-65cxx\" (UID: \"07769876-3c0c-4714-b6c7-bbd195c0d0c6\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"
Oct 30 20:27:17.490604 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:17.490551    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk9r2\" (UniqueName: \"kubernetes.io/projected/07769876-3c0c-4714-b6c7-bbd195c0d0c6-kube-api-access-tk9r2\") pod \"maas-api-94dffd7db-65cxx\" (UID: \"07769876-3c0c-4714-b6c7-bbd195c0d0c6\") " pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"
Oct 30 20:27:17.771486 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:17.771410    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"
Oct 30 20:27:18.160507 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:18.160483    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"]
Oct 30 20:27:18.162674 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:27:18.162648    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07769876_3c0c_4714_b6c7_bbd195c0d0c6.slice/crio-06e139a6259a4b07fe0b4f8ddba26bc0163f331beddfce8a694c91915e386bd1 WatchSource:0}: Error finding container 06e139a6259a4b07fe0b4f8ddba26bc0163f331beddfce8a694c91915e386bd1: Status 404 returned error can't find the container with id 06e139a6259a4b07fe0b4f8ddba26bc0163f331beddfce8a694c91915e386bd1
Oct 30 20:27:19.099438 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:19.099405    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx" event={"ID":"07769876-3c0c-4714-b6c7-bbd195c0d0c6","Type":"ContainerStarted","Data":"06e139a6259a4b07fe0b4f8ddba26bc0163f331beddfce8a694c91915e386bd1"}
Oct 30 20:27:20.104965 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:20.104931    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx" event={"ID":"07769876-3c0c-4714-b6c7-bbd195c0d0c6","Type":"ContainerStarted","Data":"affab50c837207af72315b304903179d47fcb2f4988845c65611f3eae3681af3"}
Oct 30 20:27:20.105403 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:20.105039    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"
Oct 30 20:27:20.241876 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:20.241827    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx" podStartSLOduration=2.168740161 podStartE2EDuration="3.241812767s" totalImagesPullingTime="1.073072606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:20.24067984 +0000 UTC m=+459.919308456" watchObservedRunningTime="2026-10-30 20:27:20.241812767 +0000 UTC m=+459.920441381"
Oct 30 20:27:26.114034 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:26.113999    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"
Oct 30 20:27:26.346872 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:26.346837    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"]
Oct 30 20:27:26.347340 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:26.347272    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs" podUID="4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82" containerName="maas-api" containerID="cri-o://d44afcc5f4cf6cfd8d8c9345ede764827eee1473efe205c3a8e4a1ad4be89de8" gracePeriod=30
Oct 30 20:27:26.572987 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:26.572964    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"
Oct 30 20:27:26.670100 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:26.670076    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-kube-api-access-8pl8x\" (UniqueName: \"kubernetes.io/projected/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-kube-api-access-8pl8x\") pod \"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82\" (UID: \"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82\") "
Oct 30 20:27:26.670246 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:26.670146    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-metrics-certs\") pod \"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82\" (UID: \"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82\") "
Oct 30 20:27:26.670246 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:26.670194    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-openshift-service-ca\") pod \"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82\" (UID: \"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82\") "
Oct 30 20:27:26.670246 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:26.670243    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-maas-api-tls\") pod \"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82\" (UID: \"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82\") "
Oct 30 20:27:26.670628 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:26.670600    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-openshift-service-ca" pod "4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82" (UID: "4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:27:26.672406 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:26.672379    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-metrics-certs" pod "4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82" (UID: "4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:27:26.672501 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:26.672438    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-kube-api-access-8pl8x" pod "4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82" (UID: "4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82"). InnerVolumeSpecName "kube-api-access-8pl8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:26.672501 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:26.672472    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-maas-api-tls" pod "4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82" (UID: "4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:27:26.770854 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:26.770821    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pl8x\" (UniqueName: \"kubernetes.io/projected/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-kube-api-access-8pl8x\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:26.770854 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:26.770849    4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-metrics-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:26.770854 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:26.770858    4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-openshift-service-ca\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:26.771082 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:26.770867    4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82-maas-api-tls\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:27.134295 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:27.134261    4471 generic.go:356] "Generic (PLEG): container finished" podID="4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82" containerID="d44afcc5f4cf6cfd8d8c9345ede764827eee1473efe205c3a8e4a1ad4be89de8" exitCode=0
Oct 30 20:27:27.134716 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:27.134305    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs" event={"ID":"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82","Type":"ContainerDied","Data":"d44afcc5f4cf6cfd8d8c9345ede764827eee1473efe205c3a8e4a1ad4be89de8"}
Oct 30 20:27:27.134716 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:27.134335    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"
Oct 30 20:27:27.134716 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:27.134386    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs" event={"ID":"4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82","Type":"ContainerDied","Data":"ac83fba8aee23783b8001292714d8c6e98c717f32fdaeb060f7cb66861ff860c"}
Oct 30 20:27:27.134716 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:27.134407    4471 scope.go:122] "RemoveContainer" containerID="d44afcc5f4cf6cfd8d8c9345ede764827eee1473efe205c3a8e4a1ad4be89de8"
Oct 30 20:27:27.143051 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:27.143033    4471 scope.go:122] "RemoveContainer" containerID="d44afcc5f4cf6cfd8d8c9345ede764827eee1473efe205c3a8e4a1ad4be89de8"
Oct 30 20:27:27.143291 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:27:27.143274    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d44afcc5f4cf6cfd8d8c9345ede764827eee1473efe205c3a8e4a1ad4be89de8\": container with ID starting with d44afcc5f4cf6cfd8d8c9345ede764827eee1473efe205c3a8e4a1ad4be89de8 not found: ID does not exist" containerID="d44afcc5f4cf6cfd8d8c9345ede764827eee1473efe205c3a8e4a1ad4be89de8"
Oct 30 20:27:27.143368 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:27.143299    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d44afcc5f4cf6cfd8d8c9345ede764827eee1473efe205c3a8e4a1ad4be89de8"} err="failed to get container status \"d44afcc5f4cf6cfd8d8c9345ede764827eee1473efe205c3a8e4a1ad4be89de8\": rpc error: code = NotFound desc = could not find container \"d44afcc5f4cf6cfd8d8c9345ede764827eee1473efe205c3a8e4a1ad4be89de8\": container with ID starting with d44afcc5f4cf6cfd8d8c9345ede764827eee1473efe205c3a8e4a1ad4be89de8 not found: ID does not exist"
Oct 30 20:27:27.295785 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:27.295753    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"]
Oct 30 20:27:27.344896 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:27.344867    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-5ndqs"]
Oct 30 20:27:28.429191 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:28.429157    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82" path="/var/lib/kubelet/pods/4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82/volumes"
Oct 30 20:27:30.952606 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:27:30.952570    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:27:31.028431 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:31.028410    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-xbwkj"
Oct 30 20:27:33.034544 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:27:33.034517    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69"
Oct 30 20:28:00.952225 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:28:00.952146    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:28:22.069010 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:22.068978    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-lhj95"]
Oct 30 20:28:22.069426 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:22.069400    4471 cpu_manager.go:414] "containerMap: removing container" podUID="4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82" containerName="maas-api"
Oct 30 20:28:22.069426 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:22.069415    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82" containerName="maas-api"
Oct 30 20:28:22.069510 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:22.069498    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="4985eb6f-fd0a-4dc9-951a-0ec5e6ea9d82" containerName="maas-api"
Oct 30 20:28:22.072649 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:22.072633    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lhj95"
Oct 30 20:28:22.130303 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:22.130275    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-lhj95"]
Oct 30 20:28:22.156962 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:22.156936    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce0540db-8a90-40d0-9867-73d3ee877785-cert\") pod \"llmisvc-controller-manager-6466c554d8-lhj95\" (UID: \"ce0540db-8a90-40d0-9867-73d3ee877785\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lhj95"
Oct 30 20:28:22.157126 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:22.156988    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdvmn\" (UniqueName: \"kubernetes.io/projected/ce0540db-8a90-40d0-9867-73d3ee877785-kube-api-access-rdvmn\") pod \"llmisvc-controller-manager-6466c554d8-lhj95\" (UID: \"ce0540db-8a90-40d0-9867-73d3ee877785\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lhj95"
Oct 30 20:28:22.257611 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:22.257585    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce0540db-8a90-40d0-9867-73d3ee877785-cert\") pod \"llmisvc-controller-manager-6466c554d8-lhj95\" (UID: \"ce0540db-8a90-40d0-9867-73d3ee877785\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lhj95"
Oct 30 20:28:22.257753 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:22.257640    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdvmn\" (UniqueName: \"kubernetes.io/projected/ce0540db-8a90-40d0-9867-73d3ee877785-kube-api-access-rdvmn\") pod \"llmisvc-controller-manager-6466c554d8-lhj95\" (UID: \"ce0540db-8a90-40d0-9867-73d3ee877785\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lhj95"
Oct 30 20:28:22.260105 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:22.260085    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce0540db-8a90-40d0-9867-73d3ee877785-cert\") pod \"llmisvc-controller-manager-6466c554d8-lhj95\" (UID: \"ce0540db-8a90-40d0-9867-73d3ee877785\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lhj95"
Oct 30 20:28:22.314193 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:22.314164    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdvmn\" (UniqueName: \"kubernetes.io/projected/ce0540db-8a90-40d0-9867-73d3ee877785-kube-api-access-rdvmn\") pod \"llmisvc-controller-manager-6466c554d8-lhj95\" (UID: \"ce0540db-8a90-40d0-9867-73d3ee877785\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lhj95"
Oct 30 20:28:22.384763 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:22.384744    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lhj95"
Oct 30 20:28:22.752286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:22.752262    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-lhj95"]
Oct 30 20:28:22.754742 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:28:22.754714    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0540db_8a90_40d0_9867_73d3ee877785.slice/crio-11611c2ba5e478abe7c404b239325f79a9c523429aae579c351b5eb87d3ff13d WatchSource:0}: Error finding container 11611c2ba5e478abe7c404b239325f79a9c523429aae579c351b5eb87d3ff13d: Status 404 returned error can't find the container with id 11611c2ba5e478abe7c404b239325f79a9c523429aae579c351b5eb87d3ff13d
Oct 30 20:28:23.349751 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:23.349721    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lhj95" event={"ID":"ce0540db-8a90-40d0-9867-73d3ee877785","Type":"ContainerStarted","Data":"11611c2ba5e478abe7c404b239325f79a9c523429aae579c351b5eb87d3ff13d"}
Oct 30 20:28:24.354684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:24.354652    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lhj95" event={"ID":"ce0540db-8a90-40d0-9867-73d3ee877785","Type":"ContainerStarted","Data":"69851f58a7e3db8403c4a61451f096a1ed02e6b02327d73825a42a769171c83f"}
Oct 30 20:28:24.355045 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:24.354780    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lhj95"
Oct 30 20:28:24.470497 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:24.470434    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lhj95" podStartSLOduration=1.696755282 podStartE2EDuration="2.470414103s" totalImagesPullingTime="773.658821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:28:24.470068774 +0000 UTC m=+524.148697383" watchObservedRunningTime="2026-10-30 20:28:24.470414103 +0000 UTC m=+524.149042717"
Oct 30 20:28:30.951953 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:28:30.951919    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:28:55.361214 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:55.361183    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lhj95"
Oct 30 20:28:55.527831 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:55.527791    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69"]
Oct 30 20:28:55.528367 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:55.528282    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69" podUID="2b8c9d4e-47d2-404c-b13f-ddfcd695ff15" containerName="manager" containerID="cri-o://c90d7dc570ab8725447d5a630341a43ba7b0c97d1505932a81e551eb4254e46c" gracePeriod=30
Oct 30 20:28:55.768007 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:55.767984    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69"
Oct 30 20:28:55.796403 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:55.796377    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b8c9d4e-47d2-404c-b13f-ddfcd695ff15-kube-api-access-lhgz6\" (UniqueName: \"kubernetes.io/projected/2b8c9d4e-47d2-404c-b13f-ddfcd695ff15-kube-api-access-lhgz6\") pod \"2b8c9d4e-47d2-404c-b13f-ddfcd695ff15\" (UID: \"2b8c9d4e-47d2-404c-b13f-ddfcd695ff15\") "
Oct 30 20:28:55.796505 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:55.796448    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b8c9d4e-47d2-404c-b13f-ddfcd695ff15-cert\" (UniqueName: \"kubernetes.io/secret/2b8c9d4e-47d2-404c-b13f-ddfcd695ff15-cert\") pod \"2b8c9d4e-47d2-404c-b13f-ddfcd695ff15\" (UID: \"2b8c9d4e-47d2-404c-b13f-ddfcd695ff15\") "
Oct 30 20:28:55.798381 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:55.798354    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b8c9d4e-47d2-404c-b13f-ddfcd695ff15-kube-api-access-lhgz6" pod "2b8c9d4e-47d2-404c-b13f-ddfcd695ff15" (UID: "2b8c9d4e-47d2-404c-b13f-ddfcd695ff15"). InnerVolumeSpecName "kube-api-access-lhgz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:55.798497 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:55.798482    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b8c9d4e-47d2-404c-b13f-ddfcd695ff15-cert" pod "2b8c9d4e-47d2-404c-b13f-ddfcd695ff15" (UID: "2b8c9d4e-47d2-404c-b13f-ddfcd695ff15"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:28:55.897385 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:55.897298    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhgz6\" (UniqueName: \"kubernetes.io/projected/2b8c9d4e-47d2-404c-b13f-ddfcd695ff15-kube-api-access-lhgz6\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:55.897385 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:55.897333    4471 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b8c9d4e-47d2-404c-b13f-ddfcd695ff15-cert\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:56.483711 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:56.483678    4471 generic.go:356] "Generic (PLEG): container finished" podID="2b8c9d4e-47d2-404c-b13f-ddfcd695ff15" containerID="c90d7dc570ab8725447d5a630341a43ba7b0c97d1505932a81e551eb4254e46c" exitCode=0
Oct 30 20:28:56.484098 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:56.483714    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69"
Oct 30 20:28:56.484098 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:56.483767    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69" event={"ID":"2b8c9d4e-47d2-404c-b13f-ddfcd695ff15","Type":"ContainerDied","Data":"c90d7dc570ab8725447d5a630341a43ba7b0c97d1505932a81e551eb4254e46c"}
Oct 30 20:28:56.484098 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:56.483804    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69" event={"ID":"2b8c9d4e-47d2-404c-b13f-ddfcd695ff15","Type":"ContainerDied","Data":"02bf59907633a46dd2da6c8167b0daa88cc7f6a530150159f91d40dbb7303bea"}
Oct 30 20:28:56.484098 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:56.483824    4471 scope.go:122] "RemoveContainer" containerID="c90d7dc570ab8725447d5a630341a43ba7b0c97d1505932a81e551eb4254e46c"
Oct 30 20:28:56.491830 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:56.491813    4471 scope.go:122] "RemoveContainer" containerID="c90d7dc570ab8725447d5a630341a43ba7b0c97d1505932a81e551eb4254e46c"
Oct 30 20:28:56.492116 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:28:56.492096    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c90d7dc570ab8725447d5a630341a43ba7b0c97d1505932a81e551eb4254e46c\": container with ID starting with c90d7dc570ab8725447d5a630341a43ba7b0c97d1505932a81e551eb4254e46c not found: ID does not exist" containerID="c90d7dc570ab8725447d5a630341a43ba7b0c97d1505932a81e551eb4254e46c"
Oct 30 20:28:56.492164 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:56.492125    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c90d7dc570ab8725447d5a630341a43ba7b0c97d1505932a81e551eb4254e46c"} err="failed to get container status \"c90d7dc570ab8725447d5a630341a43ba7b0c97d1505932a81e551eb4254e46c\": rpc error: code = NotFound desc = could not find container \"c90d7dc570ab8725447d5a630341a43ba7b0c97d1505932a81e551eb4254e46c\": container with ID starting with c90d7dc570ab8725447d5a630341a43ba7b0c97d1505932a81e551eb4254e46c not found: ID does not exist"
Oct 30 20:28:56.647364 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:56.647317    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69"]
Oct 30 20:28:56.697011 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:56.696952    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p4w69"]
Oct 30 20:28:58.429531 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:28:58.429498    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b8c9d4e-47d2-404c-b13f-ddfcd695ff15" path="/var/lib/kubelet/pods/2b8c9d4e-47d2-404c-b13f-ddfcd695ff15/volumes"
Oct 30 20:29:00.953455 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:29:00.953416    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:29:02.508645 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:02.508616    4471 generic.go:356] "Generic (PLEG): container finished" podID="f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b" containerID="db10ae40390eefe009b5eee2236d8cf2dc3967cf43f26a8b9c602615abd35452" exitCode=1
Oct 30 20:29:02.509040 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:02.508665    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-756d9799fc-v9l8r" event={"ID":"f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b","Type":"ContainerDied","Data":"db10ae40390eefe009b5eee2236d8cf2dc3967cf43f26a8b9c602615abd35452"}
Oct 30 20:29:02.509040 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:02.508975    4471 scope.go:122] "RemoveContainer" containerID="db10ae40390eefe009b5eee2236d8cf2dc3967cf43f26a8b9c602615abd35452"
Oct 30 20:29:03.891307 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:03.891277    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-756d9799fc-v9l8r"
Oct 30 20:29:04.517539 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:04.517504    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-756d9799fc-v9l8r" event={"ID":"f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b","Type":"ContainerStarted","Data":"b0344f9085fdcd49fcc5e58d2fdd812f86658b5be6fe62ee7b139a85bbf3b663"}
Oct 30 20:29:04.517737 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:04.517630    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-756d9799fc-v9l8r"
Oct 30 20:29:15.523936 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:15.523904    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-756d9799fc-v9l8r"
Oct 30 20:29:30.955210 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:29:30.955122    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:29:52.775005 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.774963    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"]
Oct 30 20:29:52.775629 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.775431    4471 cpu_manager.go:414] "containerMap: removing container" podUID="2b8c9d4e-47d2-404c-b13f-ddfcd695ff15" containerName="manager"
Oct 30 20:29:52.775629 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.775451    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b8c9d4e-47d2-404c-b13f-ddfcd695ff15" containerName="manager"
Oct 30 20:29:52.775629 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.775549    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b8c9d4e-47d2-404c-b13f-ddfcd695ff15" containerName="manager"
Oct 30 20:29:52.779814 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.779792    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.831197 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.831172    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-7k2d5\""
Oct 30 20:29:52.831684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.831662    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11141a83-7ed0-470a-9604-340b4d6af444-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.831733 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.831719    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11141a83-7ed0-470a-9604-340b4d6af444-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.831787 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.831773    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11141a83-7ed0-470a-9604-340b4d6af444-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.831822 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.831802    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11141a83-7ed0-470a-9604-340b4d6af444-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.831860 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.831837    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11141a83-7ed0-470a-9604-340b4d6af444-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.831860 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.831853    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11141a83-7ed0-470a-9604-340b4d6af444-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.832008 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.831992    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-724zf\" (UniqueName: \"kubernetes.io/projected/11141a83-7ed0-470a-9604-340b4d6af444-kube-api-access-724zf\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.832041 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.832020    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11141a83-7ed0-470a-9604-340b4d6af444-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.832072 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.832056    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11141a83-7ed0-470a-9604-340b4d6af444-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.886926 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.886891    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"]
Oct 30 20:29:52.932668 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.932639    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11141a83-7ed0-470a-9604-340b4d6af444-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.932847 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.932675    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11141a83-7ed0-470a-9604-340b4d6af444-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.932847 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.932710    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-724zf\" (UniqueName: \"kubernetes.io/projected/11141a83-7ed0-470a-9604-340b4d6af444-kube-api-access-724zf\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.932847 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.932734    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11141a83-7ed0-470a-9604-340b4d6af444-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.932847 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.932769    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11141a83-7ed0-470a-9604-340b4d6af444-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.932847 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.932792    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11141a83-7ed0-470a-9604-340b4d6af444-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.933053 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.932850    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11141a83-7ed0-470a-9604-340b4d6af444-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.933053 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.932875    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11141a83-7ed0-470a-9604-340b4d6af444-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.933053 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.932910    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11141a83-7ed0-470a-9604-340b4d6af444-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.933145 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.933088    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11141a83-7ed0-470a-9604-340b4d6af444-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.933392 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.933352    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11141a83-7ed0-470a-9604-340b4d6af444-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.933517 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.933426    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11141a83-7ed0-470a-9604-340b4d6af444-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.933517 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.933449    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11141a83-7ed0-470a-9604-340b4d6af444-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.933594 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.933521    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11141a83-7ed0-470a-9604-340b4d6af444-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.935463 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.935432    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11141a83-7ed0-470a-9604-340b4d6af444-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.935648 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.935631    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11141a83-7ed0-470a-9604-340b4d6af444-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.989143 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.989111    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-724zf\" (UniqueName: \"kubernetes.io/projected/11141a83-7ed0-470a-9604-340b4d6af444-kube-api-access-724zf\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:52.989431 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:52.989412    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11141a83-7ed0-470a-9604-340b4d6af444-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7458\" (UID: \"11141a83-7ed0-470a-9604-340b4d6af444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:53.091693 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:53.091621    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:53.518672 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:53.516183    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"]
Oct 30 20:29:53.522522 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:29:53.522493    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11141a83_7ed0_470a_9604_340b4d6af444.slice/crio-4d19f0f45ebef38d1094213bd112e1e5e85b77ae368230c62364c147a95e6d50 WatchSource:0}: Error finding container 4d19f0f45ebef38d1094213bd112e1e5e85b77ae368230c62364c147a95e6d50: Status 404 returned error can't find the container with id 4d19f0f45ebef38d1094213bd112e1e5e85b77ae368230c62364c147a95e6d50
Oct 30 20:29:53.710650 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:53.710622    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458" event={"ID":"11141a83-7ed0-470a-9604-340b4d6af444","Type":"ContainerStarted","Data":"4d19f0f45ebef38d1094213bd112e1e5e85b77ae368230c62364c147a95e6d50"}
Oct 30 20:29:57.514931 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:57.514897    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:29:57.515209 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:57.514973    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:29:57.515209 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:57.515017    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:29:57.730684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:57.730649    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458" event={"ID":"11141a83-7ed0-470a-9604-340b4d6af444","Type":"ContainerStarted","Data":"a8f30b3468221a9d7a1fe5ad34dc5e9a8aa432ad8ce84937795932d1c927b377"}
Oct 30 20:29:57.947240 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:57.947191    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458" podStartSLOduration=1.957474238 podStartE2EDuration="5.947176s" totalImagesPullingTime="3.989701762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:57.897163895 +0000 UTC m=+617.575792533" watchObservedRunningTime="2026-10-30 20:29:57.947176 +0000 UTC m=+617.625804614"
Oct 30 20:29:58.091987 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:58.091943    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:29:58.093681 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:58.093653    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-k7458 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 20:29:58.093805 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:58.093714    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458" podUID="11141a83-7ed0-470a-9604-340b4d6af444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 20:29:59.092999 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:59.092956    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-k7458 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 20:29:59.093487 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:29:59.093017    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458" podUID="11141a83-7ed0-470a-9604-340b4d6af444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 20:30:00.092538 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:30:00.092501    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-k7458 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 20:30:00.092691 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:30:00.092563    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458" podUID="11141a83-7ed0-470a-9604-340b4d6af444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 20:30:00.978951 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:30:00.978911    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:30:01.092191 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:30:01.092157    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-k7458 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 20:30:01.092377 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:30:01.092226    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458" podUID="11141a83-7ed0-470a-9604-340b4d6af444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 20:30:02.092375 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:30:02.092338    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-k7458 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 20:30:02.092827 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:30:02.092397    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458" podUID="11141a83-7ed0-470a-9604-340b4d6af444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 20:30:03.092770 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:30:03.092728    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-k7458 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 20:30:03.093139 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:30:03.092777    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458" podUID="11141a83-7ed0-470a-9604-340b4d6af444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 20:30:04.092882 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:30:04.092842    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-k7458 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 20:30:04.093252 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:30:04.092910    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458" podUID="11141a83-7ed0-470a-9604-340b4d6af444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 20:30:05.532860 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:30:05.532820    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:30:05.533601 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:30:05.533574    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:30:05.534627 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:30:05.534598    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458"
Oct 30 20:30:30.951957 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:30:30.951922    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:31:00.951877 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:31:00.951792    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:31:15.212024 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:15.211992    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"]
Oct 30 20:31:15.217257 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:15.217233    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"
Oct 30 20:31:15.273840 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:15.273813    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-4ee72d-serving-cert\""
Oct 30 20:31:15.280163 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:15.280136    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbl8k\" (UniqueName: \"kubernetes.io/projected/2d23e8a7-5cce-4e8b-9447-887e59719353-kube-api-access-rbl8k\") pod \"maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696\" (UID: \"2d23e8a7-5cce-4e8b-9447-887e59719353\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"
Oct 30 20:31:15.280272 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:15.280188    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d23e8a7-5cce-4e8b-9447-887e59719353-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696\" (UID: \"2d23e8a7-5cce-4e8b-9447-887e59719353\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"
Oct 30 20:31:15.280315 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:15.280277    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d23e8a7-5cce-4e8b-9447-887e59719353-maas-api-tls\") pod \"maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696\" (UID: \"2d23e8a7-5cce-4e8b-9447-887e59719353\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"
Oct 30 20:31:15.280315 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:15.280300    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d23e8a7-5cce-4e8b-9447-887e59719353-metrics-certs\") pod \"maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696\" (UID: \"2d23e8a7-5cce-4e8b-9447-887e59719353\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"
Oct 30 20:31:15.324413 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:15.324388    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"]
Oct 30 20:31:15.380863 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:15.380830    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d23e8a7-5cce-4e8b-9447-887e59719353-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696\" (UID: \"2d23e8a7-5cce-4e8b-9447-887e59719353\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"
Oct 30 20:31:15.381028 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:15.380898    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d23e8a7-5cce-4e8b-9447-887e59719353-maas-api-tls\") pod \"maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696\" (UID: \"2d23e8a7-5cce-4e8b-9447-887e59719353\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"
Oct 30 20:31:15.381028 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:15.380921    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d23e8a7-5cce-4e8b-9447-887e59719353-metrics-certs\") pod \"maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696\" (UID: \"2d23e8a7-5cce-4e8b-9447-887e59719353\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"
Oct 30 20:31:15.381028 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:15.380951    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbl8k\" (UniqueName: \"kubernetes.io/projected/2d23e8a7-5cce-4e8b-9447-887e59719353-kube-api-access-rbl8k\") pod \"maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696\" (UID: \"2d23e8a7-5cce-4e8b-9447-887e59719353\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"
Oct 30 20:31:15.381192 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:31:15.381037    4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-serving-cert: secret "maas-api-e2e-worker-w6-4ee72d-serving-cert" not found
Oct 30 20:31:15.381192 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:31:15.381101    4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d23e8a7-5cce-4e8b-9447-887e59719353-maas-api-tls podName:2d23e8a7-5cce-4e8b-9447-887e59719353 nodeName:}" failed. No retries permitted until 2026-10-30 20:31:15.88108569 +0000 UTC m=+695.559714283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d23e8a7-5cce-4e8b-9447-887e59719353-maas-api-tls") pod "maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696" (UID: "2d23e8a7-5cce-4e8b-9447-887e59719353") : secret "maas-api-e2e-worker-w6-4ee72d-serving-cert" not found
Oct 30 20:31:15.381574 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:15.381553    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d23e8a7-5cce-4e8b-9447-887e59719353-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696\" (UID: \"2d23e8a7-5cce-4e8b-9447-887e59719353\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"
Oct 30 20:31:15.383306 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:15.383288    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d23e8a7-5cce-4e8b-9447-887e59719353-metrics-certs\") pod \"maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696\" (UID: \"2d23e8a7-5cce-4e8b-9447-887e59719353\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"
Oct 30 20:31:15.443604 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:15.443577    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbl8k\" (UniqueName: \"kubernetes.io/projected/2d23e8a7-5cce-4e8b-9447-887e59719353-kube-api-access-rbl8k\") pod \"maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696\" (UID: \"2d23e8a7-5cce-4e8b-9447-887e59719353\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"
Oct 30 20:31:15.885198 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:15.885166    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d23e8a7-5cce-4e8b-9447-887e59719353-maas-api-tls\") pod \"maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696\" (UID: \"2d23e8a7-5cce-4e8b-9447-887e59719353\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"
Oct 30 20:31:15.887654 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:15.887626    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d23e8a7-5cce-4e8b-9447-887e59719353-maas-api-tls\") pod \"maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696\" (UID: \"2d23e8a7-5cce-4e8b-9447-887e59719353\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"
Oct 30 20:31:16.129217 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:16.129190    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"
Oct 30 20:31:16.499832 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:16.499756    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"]
Oct 30 20:31:16.502592 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:31:16.502566    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d23e8a7_5cce_4e8b_9447_887e59719353.slice/crio-9147987df32955868237ef401ed3c89f00f9c5bba199b94d5e46ab39e73e6220 WatchSource:0}: Error finding container 9147987df32955868237ef401ed3c89f00f9c5bba199b94d5e46ab39e73e6220: Status 404 returned error can't find the container with id 9147987df32955868237ef401ed3c89f00f9c5bba199b94d5e46ab39e73e6220
Oct 30 20:31:16.504009 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:16.503986    4471 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:31:17.053940 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:17.053904    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696" event={"ID":"2d23e8a7-5cce-4e8b-9447-887e59719353","Type":"ContainerStarted","Data":"9147987df32955868237ef401ed3c89f00f9c5bba199b94d5e46ab39e73e6220"}
Oct 30 20:31:18.059387 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:18.059349    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696" event={"ID":"2d23e8a7-5cce-4e8b-9447-887e59719353","Type":"ContainerStarted","Data":"268a3b3adacb904d0f2915ea7b64eebe13ca7a9eeee298d829b9a9f33f120c95"}
Oct 30 20:31:18.059785 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:18.059441    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"
Oct 30 20:31:18.212623 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:18.212569    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696" podStartSLOduration=2.42278323 podStartE2EDuration="3.212554427s" totalImagesPullingTime="789.771197ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:18.211731526 +0000 UTC m=+697.890360140" watchObservedRunningTime="2026-10-30 20:31:18.212554427 +0000 UTC m=+697.891183086"
Oct 30 20:31:24.069171 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:24.069144    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"
Oct 30 20:31:25.812259 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:25.812225    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"]
Oct 30 20:31:25.815422 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:25.815400    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"
Oct 30 20:31:25.867486 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:25.867458    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-ae2d38-serving-cert\""
Oct 30 20:31:25.872844 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:25.872818    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5652acc-b0ac-4755-a655-407ae5fba9a3-metrics-certs\") pod \"maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk\" (UID: \"c5652acc-b0ac-4755-a655-407ae5fba9a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"
Oct 30 20:31:25.872983 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:25.872869    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrmtv\" (UniqueName: \"kubernetes.io/projected/c5652acc-b0ac-4755-a655-407ae5fba9a3-kube-api-access-rrmtv\") pod \"maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk\" (UID: \"c5652acc-b0ac-4755-a655-407ae5fba9a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"
Oct 30 20:31:25.872983 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:25.872967    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5652acc-b0ac-4755-a655-407ae5fba9a3-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk\" (UID: \"c5652acc-b0ac-4755-a655-407ae5fba9a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"
Oct 30 20:31:25.873132 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:25.873110    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5652acc-b0ac-4755-a655-407ae5fba9a3-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk\" (UID: \"c5652acc-b0ac-4755-a655-407ae5fba9a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"
Oct 30 20:31:25.918882 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:25.918845    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"]
Oct 30 20:31:25.974435 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:25.974398    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrmtv\" (UniqueName: \"kubernetes.io/projected/c5652acc-b0ac-4755-a655-407ae5fba9a3-kube-api-access-rrmtv\") pod \"maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk\" (UID: \"c5652acc-b0ac-4755-a655-407ae5fba9a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"
Oct 30 20:31:25.974664 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:25.974458    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5652acc-b0ac-4755-a655-407ae5fba9a3-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk\" (UID: \"c5652acc-b0ac-4755-a655-407ae5fba9a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"
Oct 30 20:31:25.974664 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:25.974535    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5652acc-b0ac-4755-a655-407ae5fba9a3-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk\" (UID: \"c5652acc-b0ac-4755-a655-407ae5fba9a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"
Oct 30 20:31:25.974664 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:25.974583    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5652acc-b0ac-4755-a655-407ae5fba9a3-metrics-certs\") pod \"maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk\" (UID: \"c5652acc-b0ac-4755-a655-407ae5fba9a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"
Oct 30 20:31:25.974664 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:31:25.974596    4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-serving-cert: secret "maas-api-e2e-worker-w3-ae2d38-serving-cert" not found
Oct 30 20:31:25.974868 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:31:25.974671    4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5652acc-b0ac-4755-a655-407ae5fba9a3-maas-api-tls podName:c5652acc-b0ac-4755-a655-407ae5fba9a3 nodeName:}" failed. No retries permitted until 2026-10-30 20:31:26.474652745 +0000 UTC m=+706.153281338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c5652acc-b0ac-4755-a655-407ae5fba9a3-maas-api-tls") pod "maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk" (UID: "c5652acc-b0ac-4755-a655-407ae5fba9a3") : secret "maas-api-e2e-worker-w3-ae2d38-serving-cert" not found
Oct 30 20:31:25.975274 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:25.975252    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5652acc-b0ac-4755-a655-407ae5fba9a3-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk\" (UID: \"c5652acc-b0ac-4755-a655-407ae5fba9a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"
Oct 30 20:31:25.977145 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:25.977119    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5652acc-b0ac-4755-a655-407ae5fba9a3-metrics-certs\") pod \"maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk\" (UID: \"c5652acc-b0ac-4755-a655-407ae5fba9a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"
Oct 30 20:31:26.032362 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:26.032317    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrmtv\" (UniqueName: \"kubernetes.io/projected/c5652acc-b0ac-4755-a655-407ae5fba9a3-kube-api-access-rrmtv\") pod \"maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk\" (UID: \"c5652acc-b0ac-4755-a655-407ae5fba9a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"
Oct 30 20:31:26.479080 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:26.479046    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5652acc-b0ac-4755-a655-407ae5fba9a3-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk\" (UID: \"c5652acc-b0ac-4755-a655-407ae5fba9a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"
Oct 30 20:31:26.481454 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:26.481427    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5652acc-b0ac-4755-a655-407ae5fba9a3-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk\" (UID: \"c5652acc-b0ac-4755-a655-407ae5fba9a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"
Oct 30 20:31:26.727655 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:26.727624    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"
Oct 30 20:31:27.108057 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:27.108028    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"]
Oct 30 20:31:27.110696 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:31:27.110671    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5652acc_b0ac_4755_a655_407ae5fba9a3.slice/crio-830f1b599d59257c5c0f8486d5f45b3d7cd7938025dba3dbcf0375e12b83eead WatchSource:0}: Error finding container 830f1b599d59257c5c0f8486d5f45b3d7cd7938025dba3dbcf0375e12b83eead: Status 404 returned error can't find the container with id 830f1b599d59257c5c0f8486d5f45b3d7cd7938025dba3dbcf0375e12b83eead
Oct 30 20:31:28.102465 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:28.102381    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk" event={"ID":"c5652acc-b0ac-4755-a655-407ae5fba9a3","Type":"ContainerStarted","Data":"52b604c5f43b91311c9012b5e23276c3a59fc2c29fa5e691c4a7573ac030acc8"}
Oct 30 20:31:28.102465 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:28.102451    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk" event={"ID":"c5652acc-b0ac-4755-a655-407ae5fba9a3","Type":"ContainerStarted","Data":"830f1b599d59257c5c0f8486d5f45b3d7cd7938025dba3dbcf0375e12b83eead"}
Oct 30 20:31:28.104354 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:28.103269    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"
Oct 30 20:31:28.221154 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:28.221048    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk" podStartSLOduration=2.433792696 podStartE2EDuration="3.221033342s" totalImagesPullingTime="787.240646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:28.220052088 +0000 UTC m=+707.898680699" watchObservedRunningTime="2026-10-30 20:31:28.221033342 +0000 UTC m=+707.899661957"
Oct 30 20:31:30.611730 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:30.611692    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"]
Oct 30 20:31:30.615001 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:30.614981    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"
Oct 30 20:31:30.666807 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:30.666782    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-df031b-serving-cert\""
Oct 30 20:31:30.717753 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:30.717724    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"]
Oct 30 20:31:30.719498 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:30.719479    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccae7148-e9b3-488b-ad71-acceff9f3d9a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-df031b-6787d94787-2m276\" (UID: \"ccae7148-e9b3-488b-ad71-acceff9f3d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"
Oct 30 20:31:30.719560 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:30.719518    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccae7148-e9b3-488b-ad71-acceff9f3d9a-metrics-certs\") pod \"maas-api-e2e-worker-w2-df031b-6787d94787-2m276\" (UID: \"ccae7148-e9b3-488b-ad71-acceff9f3d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"
Oct 30 20:31:30.719619 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:30.719601    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvpzn\" (UniqueName: \"kubernetes.io/projected/ccae7148-e9b3-488b-ad71-acceff9f3d9a-kube-api-access-rvpzn\") pod \"maas-api-e2e-worker-w2-df031b-6787d94787-2m276\" (UID: \"ccae7148-e9b3-488b-ad71-acceff9f3d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"
Oct 30 20:31:30.719656 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:30.719638    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccae7148-e9b3-488b-ad71-acceff9f3d9a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-df031b-6787d94787-2m276\" (UID: \"ccae7148-e9b3-488b-ad71-acceff9f3d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"
Oct 30 20:31:30.820528 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:30.820492    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvpzn\" (UniqueName: \"kubernetes.io/projected/ccae7148-e9b3-488b-ad71-acceff9f3d9a-kube-api-access-rvpzn\") pod \"maas-api-e2e-worker-w2-df031b-6787d94787-2m276\" (UID: \"ccae7148-e9b3-488b-ad71-acceff9f3d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"
Oct 30 20:31:30.820528 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:30.820530    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccae7148-e9b3-488b-ad71-acceff9f3d9a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-df031b-6787d94787-2m276\" (UID: \"ccae7148-e9b3-488b-ad71-acceff9f3d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"
Oct 30 20:31:30.820719 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:30.820577    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccae7148-e9b3-488b-ad71-acceff9f3d9a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-df031b-6787d94787-2m276\" (UID: \"ccae7148-e9b3-488b-ad71-acceff9f3d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"
Oct 30 20:31:30.820719 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:30.820613    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccae7148-e9b3-488b-ad71-acceff9f3d9a-metrics-certs\") pod \"maas-api-e2e-worker-w2-df031b-6787d94787-2m276\" (UID: \"ccae7148-e9b3-488b-ad71-acceff9f3d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"
Oct 30 20:31:30.820810 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:31:30.820721    4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-serving-cert: secret "maas-api-e2e-worker-w2-df031b-serving-cert" not found
Oct 30 20:31:30.820810 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:31:30.820788    4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccae7148-e9b3-488b-ad71-acceff9f3d9a-maas-api-tls podName:ccae7148-e9b3-488b-ad71-acceff9f3d9a nodeName:}" failed. No retries permitted until 2026-10-30 20:31:31.320772017 +0000 UTC m=+710.999400611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ccae7148-e9b3-488b-ad71-acceff9f3d9a-maas-api-tls") pod "maas-api-e2e-worker-w2-df031b-6787d94787-2m276" (UID: "ccae7148-e9b3-488b-ad71-acceff9f3d9a") : secret "maas-api-e2e-worker-w2-df031b-serving-cert" not found
Oct 30 20:31:30.821191 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:30.821170    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccae7148-e9b3-488b-ad71-acceff9f3d9a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-df031b-6787d94787-2m276\" (UID: \"ccae7148-e9b3-488b-ad71-acceff9f3d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"
Oct 30 20:31:30.823056 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:30.823038    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccae7148-e9b3-488b-ad71-acceff9f3d9a-metrics-certs\") pod \"maas-api-e2e-worker-w2-df031b-6787d94787-2m276\" (UID: \"ccae7148-e9b3-488b-ad71-acceff9f3d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"
Oct 30 20:31:30.879876 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:30.879850    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvpzn\" (UniqueName: \"kubernetes.io/projected/ccae7148-e9b3-488b-ad71-acceff9f3d9a-kube-api-access-rvpzn\") pod \"maas-api-e2e-worker-w2-df031b-6787d94787-2m276\" (UID: \"ccae7148-e9b3-488b-ad71-acceff9f3d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"
Oct 30 20:31:30.952589 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:31:30.952556    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:31:31.326456 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:31.326372    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccae7148-e9b3-488b-ad71-acceff9f3d9a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-df031b-6787d94787-2m276\" (UID: \"ccae7148-e9b3-488b-ad71-acceff9f3d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"
Oct 30 20:31:31.328748 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:31.328729    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccae7148-e9b3-488b-ad71-acceff9f3d9a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-df031b-6787d94787-2m276\" (UID: \"ccae7148-e9b3-488b-ad71-acceff9f3d9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"
Oct 30 20:31:31.527306 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:31.527274    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"
Oct 30 20:31:31.911548 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:31.911527    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"]
Oct 30 20:31:31.914151 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:31:31.914125    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccae7148_e9b3_488b_ad71_acceff9f3d9a.slice/crio-601b5641fd6666bd712a76226312673f673be8ba8052351b6db5632de861ab10 WatchSource:0}: Error finding container 601b5641fd6666bd712a76226312673f673be8ba8052351b6db5632de861ab10: Status 404 returned error can't find the container with id 601b5641fd6666bd712a76226312673f673be8ba8052351b6db5632de861ab10
Oct 30 20:31:32.119829 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:32.119794    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276" event={"ID":"ccae7148-e9b3-488b-ad71-acceff9f3d9a","Type":"ContainerStarted","Data":"601b5641fd6666bd712a76226312673f673be8ba8052351b6db5632de861ab10"}
Oct 30 20:31:33.125269 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:33.125239    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276" event={"ID":"ccae7148-e9b3-488b-ad71-acceff9f3d9a","Type":"ContainerStarted","Data":"dd2ab414012358b8895b10913188ab28724ea3c51841a4e8a8241e1c2e36ca24"}
Oct 30 20:31:33.125644 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:33.125371    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"
Oct 30 20:31:33.241903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:33.241858    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276" podStartSLOduration=2.323050219 podStartE2EDuration="3.241844615s" totalImagesPullingTime="918.794396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:33.239742626 +0000 UTC m=+712.918371241" watchObservedRunningTime="2026-10-30 20:31:33.241844615 +0000 UTC m=+712.920473229"
Oct 30 20:31:35.116399 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:35.116370    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"
Oct 30 20:31:39.133533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:39.133499    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"
Oct 30 20:31:58.787062 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.787029    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"]
Oct 30 20:31:58.790432 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.790411    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:31:58.858827 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.858797    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-home\") pod \"simulator-w6-4ee72d-kserve-5f7d99b6db-77h46\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") " pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:31:58.858827 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.858832    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-tmp-dir\") pod \"simulator-w6-4ee72d-kserve-5f7d99b6db-77h46\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") " pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:31:58.859033 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.858865    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztp59\" (UniqueName: \"kubernetes.io/projected/13896eb7-f9cc-4cf2-9364-b723651037af-kube-api-access-ztp59\") pod \"simulator-w6-4ee72d-kserve-5f7d99b6db-77h46\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") " pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:31:58.859077 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.859037    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-model-cache\") pod \"simulator-w6-4ee72d-kserve-5f7d99b6db-77h46\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") " pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:31:58.859112 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.859075    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-dshm\") pod \"simulator-w6-4ee72d-kserve-5f7d99b6db-77h46\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") " pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:31:58.859155 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.859117    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13896eb7-f9cc-4cf2-9364-b723651037af-tls-certs\") pod \"simulator-w6-4ee72d-kserve-5f7d99b6db-77h46\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") " pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:31:58.863710 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.863689    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-4ee72d\"/\"kube-root-ca.crt\""
Oct 30 20:31:58.891892 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.891861    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-4ee72d\"/\"simulator-w6-4ee72d-kserve-self-signed-certs\""
Oct 30 20:31:58.891892 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.891871    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-4ee72d\"/\"openshift-service-ca.crt\""
Oct 30 20:31:58.892084 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.891953    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-4ee72d\"/\"default-dockercfg-wgmdp\""
Oct 30 20:31:58.893375 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.893352    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"]
Oct 30 20:31:58.959523 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.959494    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-model-cache\") pod \"simulator-w6-4ee72d-kserve-5f7d99b6db-77h46\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") " pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:31:58.959523 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.959524    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-dshm\") pod \"simulator-w6-4ee72d-kserve-5f7d99b6db-77h46\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") " pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:31:58.959734 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.959543    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13896eb7-f9cc-4cf2-9364-b723651037af-tls-certs\") pod \"simulator-w6-4ee72d-kserve-5f7d99b6db-77h46\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") " pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:31:58.959734 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.959566    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-home\") pod \"simulator-w6-4ee72d-kserve-5f7d99b6db-77h46\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") " pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:31:58.959734 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.959584    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-tmp-dir\") pod \"simulator-w6-4ee72d-kserve-5f7d99b6db-77h46\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") " pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:31:58.959734 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.959642    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztp59\" (UniqueName: \"kubernetes.io/projected/13896eb7-f9cc-4cf2-9364-b723651037af-kube-api-access-ztp59\") pod \"simulator-w6-4ee72d-kserve-5f7d99b6db-77h46\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") " pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:31:58.960034 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.960001    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-model-cache\") pod \"simulator-w6-4ee72d-kserve-5f7d99b6db-77h46\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") " pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:31:58.960034 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.960013    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-home\") pod \"simulator-w6-4ee72d-kserve-5f7d99b6db-77h46\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") " pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:31:58.960208 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.960142    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-tmp-dir\") pod \"simulator-w6-4ee72d-kserve-5f7d99b6db-77h46\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") " pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:31:58.962003 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.961978    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-dshm\") pod \"simulator-w6-4ee72d-kserve-5f7d99b6db-77h46\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") " pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:31:58.962220 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:58.962205    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13896eb7-f9cc-4cf2-9364-b723651037af-tls-certs\") pod \"simulator-w6-4ee72d-kserve-5f7d99b6db-77h46\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") " pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:31:59.016378 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:59.016350    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztp59\" (UniqueName: \"kubernetes.io/projected/13896eb7-f9cc-4cf2-9364-b723651037af-kube-api-access-ztp59\") pod \"simulator-w6-4ee72d-kserve-5f7d99b6db-77h46\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") " pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:31:59.120781 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:59.120714    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:31:59.492931 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:31:59.492904    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"]
Oct 30 20:31:59.495305 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:31:59.495278    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13896eb7_f9cc_4cf2_9364_b723651037af.slice/crio-17dd6cc59d7dd458d783c423be43817fd1f3d634c898e0fa5d85e40fe4ebea91 WatchSource:0}: Error finding container 17dd6cc59d7dd458d783c423be43817fd1f3d634c898e0fa5d85e40fe4ebea91: Status 404 returned error can't find the container with id 17dd6cc59d7dd458d783c423be43817fd1f3d634c898e0fa5d85e40fe4ebea91
Oct 30 20:32:00.247155 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:00.247114    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46" event={"ID":"13896eb7-f9cc-4cf2-9364-b723651037af","Type":"ContainerStarted","Data":"17dd6cc59d7dd458d783c423be43817fd1f3d634c898e0fa5d85e40fe4ebea91"}
Oct 30 20:32:00.954954 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:32:00.954917    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:32:01.253420 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:01.253334    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46" event={"ID":"13896eb7-f9cc-4cf2-9364-b723651037af","Type":"ContainerStarted","Data":"a6e63d232f90d0c0755e5c950af33ed9ff0a5aa077dc4d75976d435640860ab9"}
Oct 30 20:32:01.366985 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:01.366926    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46" podStartSLOduration=1.758066261 podStartE2EDuration="3.366908246s" totalImagesPullingTime="1.608841985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:01.365526201 +0000 UTC m=+741.044154816" watchObservedRunningTime="2026-10-30 20:32:01.366908246 +0000 UTC m=+741.045536862"
Oct 30 20:32:02.121543 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:02.121512    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:32:09.120992 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:09.120953    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:32:09.137813 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:09.137782    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:32:09.298751 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:09.298720    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:32:24.205820 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.205736    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"]
Oct 30 20:32:24.210046 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.210028    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:24.255070 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.255032    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-tmp-dir\") pod \"simulator-w0-0a78ac-kserve-859c9547ff-m5f8b\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") " pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:24.255268 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.255233    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45j64\" (UniqueName: \"kubernetes.io/projected/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-kube-api-access-45j64\") pod \"simulator-w0-0a78ac-kserve-859c9547ff-m5f8b\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") " pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:24.255356 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.255272    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-tls-certs\") pod \"simulator-w0-0a78ac-kserve-859c9547ff-m5f8b\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") " pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:24.255356 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.255296    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-dshm\") pod \"simulator-w0-0a78ac-kserve-859c9547ff-m5f8b\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") " pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:24.255444 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.255415    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-home\") pod \"simulator-w0-0a78ac-kserve-859c9547ff-m5f8b\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") " pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:24.255482 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.255447    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-model-cache\") pod \"simulator-w0-0a78ac-kserve-859c9547ff-m5f8b\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") " pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:24.265775 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.265747    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0a78ac\"/\"openshift-service-ca.crt\""
Oct 30 20:32:24.311621 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.311590    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0a78ac\"/\"default-dockercfg-h5hv9\""
Oct 30 20:32:24.311824 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.311629    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0a78ac\"/\"kube-root-ca.crt\""
Oct 30 20:32:24.311824 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.311639    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0a78ac\"/\"simulator-w0-0a78ac-kserve-self-signed-certs\""
Oct 30 20:32:24.314200 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.314171    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"]
Oct 30 20:32:24.356138 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.356104    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45j64\" (UniqueName: \"kubernetes.io/projected/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-kube-api-access-45j64\") pod \"simulator-w0-0a78ac-kserve-859c9547ff-m5f8b\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") " pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:24.356414 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.356390    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-tls-certs\") pod \"simulator-w0-0a78ac-kserve-859c9547ff-m5f8b\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") " pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:24.356539 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.356522    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-dshm\") pod \"simulator-w0-0a78ac-kserve-859c9547ff-m5f8b\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") " pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:24.356693 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.356678    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-home\") pod \"simulator-w0-0a78ac-kserve-859c9547ff-m5f8b\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") " pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:24.356801 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.356788    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-model-cache\") pod \"simulator-w0-0a78ac-kserve-859c9547ff-m5f8b\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") " pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:24.356902 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.356889    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-tmp-dir\") pod \"simulator-w0-0a78ac-kserve-859c9547ff-m5f8b\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") " pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:24.357504 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.357481    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-tmp-dir\") pod \"simulator-w0-0a78ac-kserve-859c9547ff-m5f8b\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") " pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:24.357944 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.357920    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-home\") pod \"simulator-w0-0a78ac-kserve-859c9547ff-m5f8b\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") " pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:24.358188 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.358168    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-model-cache\") pod \"simulator-w0-0a78ac-kserve-859c9547ff-m5f8b\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") " pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:24.360503 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.360478    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-dshm\") pod \"simulator-w0-0a78ac-kserve-859c9547ff-m5f8b\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") " pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:24.360681 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.360660    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-tls-certs\") pod \"simulator-w0-0a78ac-kserve-859c9547ff-m5f8b\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") " pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:24.414595 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.414557    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45j64\" (UniqueName: \"kubernetes.io/projected/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-kube-api-access-45j64\") pod \"simulator-w0-0a78ac-kserve-859c9547ff-m5f8b\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") " pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:24.520490 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.520386    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:24.897476 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:24.897438    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"]
Oct 30 20:32:24.900309 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:32:24.900282    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe1e1b3_101a_4b20_a7e9_f1c287fa12cf.slice/crio-e169789f3765034cd1fd126545f77455f0a548179164157fb4288595dae01b66 WatchSource:0}: Error finding container e169789f3765034cd1fd126545f77455f0a548179164157fb4288595dae01b66: Status 404 returned error can't find the container with id e169789f3765034cd1fd126545f77455f0a548179164157fb4288595dae01b66
Oct 30 20:32:25.362919 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:25.362824    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b" event={"ID":"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf","Type":"ContainerStarted","Data":"c5944b541c6f90b78ae628e8fb2bc04667b6d6264eac9a32a71486d53b378522"}
Oct 30 20:32:25.362919 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:25.362870    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b" event={"ID":"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf","Type":"ContainerStarted","Data":"e169789f3765034cd1fd126545f77455f0a548179164157fb4288595dae01b66"}
Oct 30 20:32:25.479471 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:25.479423    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b" podStartSLOduration=1.479407019 podStartE2EDuration="1.479407019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:25.477717101 +0000 UTC m=+765.156345730" watchObservedRunningTime="2026-10-30 20:32:25.479407019 +0000 UTC m=+765.158035630"
Oct 30 20:32:25.521514 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:25.521474    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:30.954641 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:32:30.954601    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:32:34.520637 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:34.520594    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:34.533203 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:34.533170    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:35.415906 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:35.415878    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:32:47.381676 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.381640    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"]
Oct 30 20:32:47.386509 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.386490    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:47.452950 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.452915    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-dshm\") pod \"simulator-w2-df031b-kserve-7d847db4b5-4l524\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") " pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:47.452950 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.452957    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-home\") pod \"simulator-w2-df031b-kserve-7d847db4b5-4l524\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") " pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:47.453145 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.452973    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-model-cache\") pod \"simulator-w2-df031b-kserve-7d847db4b5-4l524\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") " pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:47.453145 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.453056    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7tvc\" (UniqueName: \"kubernetes.io/projected/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-kube-api-access-m7tvc\") pod \"simulator-w2-df031b-kserve-7d847db4b5-4l524\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") " pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:47.453220 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.453202    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-tmp-dir\") pod \"simulator-w2-df031b-kserve-7d847db4b5-4l524\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") " pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:47.453315 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.453300    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-tls-certs\") pod \"simulator-w2-df031b-kserve-7d847db4b5-4l524\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") " pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:47.489377 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.489312    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-df031b\"/\"openshift-service-ca.crt\""
Oct 30 20:32:47.489377 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.489349    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-df031b\"/\"default-dockercfg-7xnk5\""
Oct 30 20:32:47.489642 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.489560    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-df031b\"/\"kube-root-ca.crt\""
Oct 30 20:32:47.489642 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.489585    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-df031b\"/\"simulator-w2-df031b-kserve-self-signed-certs\""
Oct 30 20:32:47.491095 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.491075    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"]
Oct 30 20:32:47.554256 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.554205    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-tmp-dir\") pod \"simulator-w2-df031b-kserve-7d847db4b5-4l524\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") " pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:47.554512 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.554364    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-tls-certs\") pod \"simulator-w2-df031b-kserve-7d847db4b5-4l524\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") " pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:47.554512 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.554416    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-dshm\") pod \"simulator-w2-df031b-kserve-7d847db4b5-4l524\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") " pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:47.554512 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.554467    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-home\") pod \"simulator-w2-df031b-kserve-7d847db4b5-4l524\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") " pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:47.554512 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.554488    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-model-cache\") pod \"simulator-w2-df031b-kserve-7d847db4b5-4l524\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") " pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:47.554512 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.554510    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7tvc\" (UniqueName: \"kubernetes.io/projected/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-kube-api-access-m7tvc\") pod \"simulator-w2-df031b-kserve-7d847db4b5-4l524\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") " pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:47.554919 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.554894    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-tmp-dir\") pod \"simulator-w2-df031b-kserve-7d847db4b5-4l524\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") " pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:47.554919 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.554913    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-model-cache\") pod \"simulator-w2-df031b-kserve-7d847db4b5-4l524\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") " pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:47.555046 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.554998    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-home\") pod \"simulator-w2-df031b-kserve-7d847db4b5-4l524\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") " pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:47.556978 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.556957    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-dshm\") pod \"simulator-w2-df031b-kserve-7d847db4b5-4l524\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") " pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:47.557128 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.557109    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-tls-certs\") pod \"simulator-w2-df031b-kserve-7d847db4b5-4l524\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") " pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:47.612064 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.612024    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7tvc\" (UniqueName: \"kubernetes.io/projected/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-kube-api-access-m7tvc\") pod \"simulator-w2-df031b-kserve-7d847db4b5-4l524\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") " pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:47.697966 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:47.697858    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:48.095275 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:48.095248    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"]
Oct 30 20:32:48.096498 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:32:48.096468    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31cc010d_9e1f_42c2_a736_0ceacbedb8c9.slice/crio-4ec6d1c989e350c829f6cc6c18ba2ad68043e2f67db6349ca21c52a5539b1236 WatchSource:0}: Error finding container 4ec6d1c989e350c829f6cc6c18ba2ad68043e2f67db6349ca21c52a5539b1236: Status 404 returned error can't find the container with id 4ec6d1c989e350c829f6cc6c18ba2ad68043e2f67db6349ca21c52a5539b1236
Oct 30 20:32:48.466090 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:48.466047    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524" event={"ID":"31cc010d-9e1f-42c2-a736-0ceacbedb8c9","Type":"ContainerStarted","Data":"205eec1bf53a4b6425a8710fc9ee8bc97cf1a836d0798654d7518170baf1e69f"}
Oct 30 20:32:48.466090 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:48.466096    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524" event={"ID":"31cc010d-9e1f-42c2-a736-0ceacbedb8c9","Type":"ContainerStarted","Data":"4ec6d1c989e350c829f6cc6c18ba2ad68043e2f67db6349ca21c52a5539b1236"}
Oct 30 20:32:48.586302 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:48.586235    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524" podStartSLOduration=1.586209115 podStartE2EDuration="1.586209115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:48.583266014 +0000 UTC m=+788.261894629" watchObservedRunningTime="2026-10-30 20:32:48.586209115 +0000 UTC m=+788.264837731"
Oct 30 20:32:48.702776 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:48.702725    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:57.698273 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:57.698228    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:57.710960 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:57.710922    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:32:58.519308 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:32:58.519268    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:33:00.954365 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:33:00.954304    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:33:17.791792 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:17.791756    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"]
Oct 30 20:33:17.795507 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:17.795488    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:17.847363 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:17.847300    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5f9b95\"/\"kube-root-ca.crt\""
Oct 30 20:33:17.896700 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:17.896663    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5f9b95\"/\"openshift-service-ca.crt\""
Oct 30 20:33:17.896700 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:17.896663    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f9b95\"/\"default-dockercfg-6bjt2\""
Oct 30 20:33:17.896907 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:17.896778    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f9b95\"/\"simulator-w1-5f9b95-kserve-self-signed-certs\""
Oct 30 20:33:17.898649 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:17.898628    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"]
Oct 30 20:33:17.927457 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:17.927419    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dcbb5f5-93f4-4210-8274-2278fb9924ae-tls-certs\") pod \"simulator-w1-5f9b95-kserve-644d49b84b-sj5lh\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") " pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:17.927634 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:17.927493    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-home\") pod \"simulator-w1-5f9b95-kserve-644d49b84b-sj5lh\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") " pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:17.927681 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:17.927639    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-model-cache\") pod \"simulator-w1-5f9b95-kserve-644d49b84b-sj5lh\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") " pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:17.927681 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:17.927667    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-tmp-dir\") pod \"simulator-w1-5f9b95-kserve-644d49b84b-sj5lh\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") " pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:17.927767 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:17.927686    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thq2k\" (UniqueName: \"kubernetes.io/projected/3dcbb5f5-93f4-4210-8274-2278fb9924ae-kube-api-access-thq2k\") pod \"simulator-w1-5f9b95-kserve-644d49b84b-sj5lh\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") " pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:17.927767 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:17.927746    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-dshm\") pod \"simulator-w1-5f9b95-kserve-644d49b84b-sj5lh\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") " pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:18.028562 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:18.028531    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dcbb5f5-93f4-4210-8274-2278fb9924ae-tls-certs\") pod \"simulator-w1-5f9b95-kserve-644d49b84b-sj5lh\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") " pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:18.028771 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:18.028580    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-home\") pod \"simulator-w1-5f9b95-kserve-644d49b84b-sj5lh\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") " pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:18.028771 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:18.028631    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-model-cache\") pod \"simulator-w1-5f9b95-kserve-644d49b84b-sj5lh\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") " pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:18.028771 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:18.028647    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-tmp-dir\") pod \"simulator-w1-5f9b95-kserve-644d49b84b-sj5lh\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") " pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:18.028771 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:18.028665    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thq2k\" (UniqueName: \"kubernetes.io/projected/3dcbb5f5-93f4-4210-8274-2278fb9924ae-kube-api-access-thq2k\") pod \"simulator-w1-5f9b95-kserve-644d49b84b-sj5lh\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") " pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:18.028771 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:18.028683    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-dshm\") pod \"simulator-w1-5f9b95-kserve-644d49b84b-sj5lh\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") " pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:18.029159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:18.029132    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-home\") pod \"simulator-w1-5f9b95-kserve-644d49b84b-sj5lh\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") " pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:18.029219 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:18.029170    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-tmp-dir\") pod \"simulator-w1-5f9b95-kserve-644d49b84b-sj5lh\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") " pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:18.029254 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:18.029214    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-model-cache\") pod \"simulator-w1-5f9b95-kserve-644d49b84b-sj5lh\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") " pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:18.031250 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:18.031227    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-dshm\") pod \"simulator-w1-5f9b95-kserve-644d49b84b-sj5lh\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") " pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:18.031395 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:18.031256    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dcbb5f5-93f4-4210-8274-2278fb9924ae-tls-certs\") pod \"simulator-w1-5f9b95-kserve-644d49b84b-sj5lh\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") " pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:18.086491 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:18.086408    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thq2k\" (UniqueName: \"kubernetes.io/projected/3dcbb5f5-93f4-4210-8274-2278fb9924ae-kube-api-access-thq2k\") pod \"simulator-w1-5f9b95-kserve-644d49b84b-sj5lh\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") " pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:18.107448 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:18.107405    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:18.482874 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:18.482772    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"]
Oct 30 20:33:18.485239 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:33:18.485212    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3dcbb5f5_93f4_4210_8274_2278fb9924ae.slice/crio-491a7b794191d0a14bd9e6853b3a372e7dedc0847bf1a779be75b9df69a5e7c0 WatchSource:0}: Error finding container 491a7b794191d0a14bd9e6853b3a372e7dedc0847bf1a779be75b9df69a5e7c0: Status 404 returned error can't find the container with id 491a7b794191d0a14bd9e6853b3a372e7dedc0847bf1a779be75b9df69a5e7c0
Oct 30 20:33:18.601779 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:18.601720    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh" event={"ID":"3dcbb5f5-93f4-4210-8274-2278fb9924ae","Type":"ContainerStarted","Data":"a954e1fe27ee433244cd0569e394ac1992eba114052efc8a5097fec568fad8f4"}
Oct 30 20:33:18.601779 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:18.601773    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh" event={"ID":"3dcbb5f5-93f4-4210-8274-2278fb9924ae","Type":"ContainerStarted","Data":"491a7b794191d0a14bd9e6853b3a372e7dedc0847bf1a779be75b9df69a5e7c0"}
Oct 30 20:33:19.108133 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:19.108084    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:28.107541 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:28.107504    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:28.120238 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:28.120199    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:28.235649 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:28.235597    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh" podStartSLOduration=11.235584161 podStartE2EDuration="11.235584161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:18.727205122 +0000 UTC m=+818.405833737" watchObservedRunningTime="2026-10-30 20:33:28.235584161 +0000 UTC m=+827.914212777"
Oct 30 20:33:28.655715 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:28.655680    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:33:30.954076 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:33:30.954037    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:33:53.796187 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.796107    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"]
Oct 30 20:33:53.800144 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.800123    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:33:53.861276 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.861248    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-dshm\") pod \"premium-w3-ae2d38-kserve-6577db47cd-45ns4\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") " pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:33:53.861448 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.861314    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-tmp-dir\") pod \"premium-w3-ae2d38-kserve-6577db47cd-45ns4\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") " pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:33:53.861554 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.861532    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-model-cache\") pod \"premium-w3-ae2d38-kserve-6577db47cd-45ns4\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") " pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:33:53.861613 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.861564    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-tls-certs\") pod \"premium-w3-ae2d38-kserve-6577db47cd-45ns4\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") " pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:33:53.861613 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.861593    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-home\") pod \"premium-w3-ae2d38-kserve-6577db47cd-45ns4\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") " pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:33:53.861733 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.861713    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wnt7\" (UniqueName: \"kubernetes.io/projected/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-kube-api-access-4wnt7\") pod \"premium-w3-ae2d38-kserve-6577db47cd-45ns4\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") " pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:33:53.901283 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.901256    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ae2d38\"/\"openshift-service-ca.crt\""
Oct 30 20:33:53.901283 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.901265    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ae2d38\"/\"kube-root-ca.crt\""
Oct 30 20:33:53.901489 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.901350    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ae2d38\"/\"default-dockercfg-4cgd2\""
Oct 30 20:33:53.901489 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.901406    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ae2d38\"/\"premium-w3-ae2d38-kserve-self-signed-certs\""
Oct 30 20:33:53.903200 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.903176    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"]
Oct 30 20:33:53.962512 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.962475    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wnt7\" (UniqueName: \"kubernetes.io/projected/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-kube-api-access-4wnt7\") pod \"premium-w3-ae2d38-kserve-6577db47cd-45ns4\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") " pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:33:53.963582 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.963559    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-dshm\") pod \"premium-w3-ae2d38-kserve-6577db47cd-45ns4\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") " pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:33:53.963708 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.963626    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-tmp-dir\") pod \"premium-w3-ae2d38-kserve-6577db47cd-45ns4\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") " pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:33:53.963708 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.963680    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-model-cache\") pod \"premium-w3-ae2d38-kserve-6577db47cd-45ns4\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") " pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:33:53.963708 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.963699    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-tls-certs\") pod \"premium-w3-ae2d38-kserve-6577db47cd-45ns4\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") " pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:33:53.963864 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.963719    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-home\") pod \"premium-w3-ae2d38-kserve-6577db47cd-45ns4\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") " pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:33:53.964051 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.964031    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-tmp-dir\") pod \"premium-w3-ae2d38-kserve-6577db47cd-45ns4\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") " pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:33:53.964294 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.964274    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-home\") pod \"premium-w3-ae2d38-kserve-6577db47cd-45ns4\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") " pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:33:53.964409 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.964360    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-model-cache\") pod \"premium-w3-ae2d38-kserve-6577db47cd-45ns4\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") " pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:33:53.965903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.965886    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-dshm\") pod \"premium-w3-ae2d38-kserve-6577db47cd-45ns4\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") " pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:33:53.966260 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:53.966244    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-tls-certs\") pod \"premium-w3-ae2d38-kserve-6577db47cd-45ns4\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") " pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:33:54.022149 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:54.022113    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wnt7\" (UniqueName: \"kubernetes.io/projected/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-kube-api-access-4wnt7\") pod \"premium-w3-ae2d38-kserve-6577db47cd-45ns4\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") " pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:33:54.132193 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:54.132144    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:33:54.524579 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:54.524557    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"]
Oct 30 20:33:54.527014 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:33:54.526985    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ec28e09_b926_4e73_8f40_2f1ab3a41df8.slice/crio-933e3fcebe8c2f395fdaee75972d53362b225462bfd4a7883bf98df6cf96289b WatchSource:0}: Error finding container 933e3fcebe8c2f395fdaee75972d53362b225462bfd4a7883bf98df6cf96289b: Status 404 returned error can't find the container with id 933e3fcebe8c2f395fdaee75972d53362b225462bfd4a7883bf98df6cf96289b
Oct 30 20:33:54.765221 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:54.765178    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4" event={"ID":"0ec28e09-b926-4e73-8f40-2f1ab3a41df8","Type":"ContainerStarted","Data":"ef2bbee2f623f68975ea752df53ea4fc676723306f18add0cfda14046d8fc268"}
Oct 30 20:33:54.765221 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:54.765225    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4" event={"ID":"0ec28e09-b926-4e73-8f40-2f1ab3a41df8","Type":"ContainerStarted","Data":"933e3fcebe8c2f395fdaee75972d53362b225462bfd4a7883bf98df6cf96289b"}
Oct 30 20:33:54.883397 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:54.883290    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4" podStartSLOduration=1.883276882 podStartE2EDuration="1.883276882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:54.881599694 +0000 UTC m=+854.560228302" watchObservedRunningTime="2026-10-30 20:33:54.883276882 +0000 UTC m=+854.561905518"
Oct 30 20:33:55.132936 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:33:55.132903    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:34:00.953803 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:34:00.953764    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:34:04.132486 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:04.132453    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:34:04.144649 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:04.144627    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:34:04.821603 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:04.821576    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:34:28.904672 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:28.904634    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"]
Oct 30 20:34:28.907921 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:28.907904    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:28.959815 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:28.959785    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-a78bf4\"/\"kube-root-ca.crt\""
Oct 30 20:34:28.988527 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:28.988485    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-dshm\") pod \"premium-w5-a78bf4-kserve-66f5656669-5spd7\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") " pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:28.988696 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:28.988599    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcls7\" (UniqueName: \"kubernetes.io/projected/02b9bb13-c775-44a5-b034-563359097687-kube-api-access-qcls7\") pod \"premium-w5-a78bf4-kserve-66f5656669-5spd7\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") " pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:28.988696 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:28.988640    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-home\") pod \"premium-w5-a78bf4-kserve-66f5656669-5spd7\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") " pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:28.988766 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:28.988696    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02b9bb13-c775-44a5-b034-563359097687-tls-certs\") pod \"premium-w5-a78bf4-kserve-66f5656669-5spd7\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") " pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:28.988766 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:28.988726    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-model-cache\") pod \"premium-w5-a78bf4-kserve-66f5656669-5spd7\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") " pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:28.988827 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:28.988768    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-tmp-dir\") pod \"premium-w5-a78bf4-kserve-66f5656669-5spd7\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") " pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:29.009113 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:29.009077    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-a78bf4\"/\"openshift-service-ca.crt\""
Oct 30 20:34:29.009269 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:29.009175    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a78bf4\"/\"premium-w5-a78bf4-kserve-self-signed-certs\""
Oct 30 20:34:29.009269 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:29.009235    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a78bf4\"/\"default-dockercfg-999rs\""
Oct 30 20:34:29.010963 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:29.010946    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"]
Oct 30 20:34:29.089418 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:29.089377    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-tmp-dir\") pod \"premium-w5-a78bf4-kserve-66f5656669-5spd7\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") " pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:29.089418 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:29.089425    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-dshm\") pod \"premium-w5-a78bf4-kserve-66f5656669-5spd7\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") " pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:29.089684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:29.089470    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcls7\" (UniqueName: \"kubernetes.io/projected/02b9bb13-c775-44a5-b034-563359097687-kube-api-access-qcls7\") pod \"premium-w5-a78bf4-kserve-66f5656669-5spd7\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") " pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:29.089684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:29.089500    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-home\") pod \"premium-w5-a78bf4-kserve-66f5656669-5spd7\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") " pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:29.089684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:29.089522    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02b9bb13-c775-44a5-b034-563359097687-tls-certs\") pod \"premium-w5-a78bf4-kserve-66f5656669-5spd7\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") " pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:29.089684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:29.089545    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-model-cache\") pod \"premium-w5-a78bf4-kserve-66f5656669-5spd7\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") " pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:29.089924 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:29.089901    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-tmp-dir\") pod \"premium-w5-a78bf4-kserve-66f5656669-5spd7\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") " pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:29.089924 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:29.089916    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-home\") pod \"premium-w5-a78bf4-kserve-66f5656669-5spd7\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") " pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:29.090037 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:29.089990    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-model-cache\") pod \"premium-w5-a78bf4-kserve-66f5656669-5spd7\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") " pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:29.092061 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:29.092037    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-dshm\") pod \"premium-w5-a78bf4-kserve-66f5656669-5spd7\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") " pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:29.092354 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:29.092316    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02b9bb13-c775-44a5-b034-563359097687-tls-certs\") pod \"premium-w5-a78bf4-kserve-66f5656669-5spd7\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") " pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:29.146661 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:29.146628    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcls7\" (UniqueName: \"kubernetes.io/projected/02b9bb13-c775-44a5-b034-563359097687-kube-api-access-qcls7\") pod \"premium-w5-a78bf4-kserve-66f5656669-5spd7\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") " pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:29.220643 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:29.220546    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:29.627120 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:29.627096    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"]
Oct 30 20:34:29.629209 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:34:29.629180    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02b9bb13_c775_44a5_b034_563359097687.slice/crio-45482886c79faccc7986653b080aa18520de2cd8869901c4c7fcf414b82c04e6 WatchSource:0}: Error finding container 45482886c79faccc7986653b080aa18520de2cd8869901c4c7fcf414b82c04e6: Status 404 returned error can't find the container with id 45482886c79faccc7986653b080aa18520de2cd8869901c4c7fcf414b82c04e6
Oct 30 20:34:29.925691 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:29.925602    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7" event={"ID":"02b9bb13-c775-44a5-b034-563359097687","Type":"ContainerStarted","Data":"f3099c762c9e2b05eee7d9271758075e246c2861d1a6c2cbce523ab0a84c90a6"}
Oct 30 20:34:29.925691 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:29.925646    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7" event={"ID":"02b9bb13-c775-44a5-b034-563359097687","Type":"ContainerStarted","Data":"45482886c79faccc7986653b080aa18520de2cd8869901c4c7fcf414b82c04e6"}
Oct 30 20:34:30.044509 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:30.044452    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7" podStartSLOduration=2.044432893 podStartE2EDuration="2.044432893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:34:30.04257098 +0000 UTC m=+889.721199596" watchObservedRunningTime="2026-10-30 20:34:30.044432893 +0000 UTC m=+889.723061508"
Oct 30 20:34:30.220769 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:30.220679    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:30.955218 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:34:30.955179    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:34:39.221155 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:39.221114    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:39.234033 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:39.234011    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:34:39.982295 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:34:39.982265    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:35:00.955798 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:35:00.955761    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:35:30.953200 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:35:30.953118    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:36:00.953568 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:36:00.953532    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:36:11.343085 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.343049    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"]
Oct 30 20:36:11.346566 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.346547    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:11.447770 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.447744    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0a78ac\"/\"distinct-w0-0a78ac-kserve-self-signed-certs\""
Oct 30 20:36:11.449521 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.449484    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"]
Oct 30 20:36:11.468641 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.468600    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-dshm\") pod \"distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:11.468641 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.468641    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-tmp-dir\") pod \"distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:11.468874 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.468766    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-model-cache\") pod \"distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:11.468874 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.468813    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ffv6\" (UniqueName: \"kubernetes.io/projected/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-kube-api-access-7ffv6\") pod \"distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:11.468874 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.468841    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-home\") pod \"distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:11.468874 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.468861    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-tls-certs\") pod \"distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:11.570215 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.570169    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-dshm\") pod \"distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:11.570215 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.570222    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-tmp-dir\") pod \"distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:11.570515 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.570272    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-model-cache\") pod \"distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:11.570515 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.570340    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ffv6\" (UniqueName: \"kubernetes.io/projected/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-kube-api-access-7ffv6\") pod \"distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:11.570515 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.570374    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-home\") pod \"distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:11.570515 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.570407    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-tls-certs\") pod \"distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:11.570757 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.570733    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-tmp-dir\") pod \"distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:11.570797 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.570760    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-model-cache\") pod \"distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:11.570829 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.570813    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-home\") pod \"distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:11.572730 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.572601    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-dshm\") pod \"distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:11.573050 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.573030    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-tls-certs\") pod \"distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:11.627146 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.627109    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ffv6\" (UniqueName: \"kubernetes.io/projected/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-kube-api-access-7ffv6\") pod \"distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:11.658785 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:11.658751    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:12.062346 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:12.062238    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"]
Oct 30 20:36:12.066158 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:36:12.066125    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5bf5e57f_4c83_4c4b_9771_5f78d48d53ea.slice/crio-5c8a17a2e4a3416b9a1df395838abd696c8a4ae16c6c7719075a59eb32e9baaf WatchSource:0}: Error finding container 5c8a17a2e4a3416b9a1df395838abd696c8a4ae16c6c7719075a59eb32e9baaf: Status 404 returned error can't find the container with id 5c8a17a2e4a3416b9a1df395838abd696c8a4ae16c6c7719075a59eb32e9baaf
Oct 30 20:36:12.386818 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:12.386777    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht" event={"ID":"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea","Type":"ContainerStarted","Data":"43f5dce5f0b4055f71c0ca1f00de1b629d5374465f813a13ef67d5ec52f107b9"}
Oct 30 20:36:12.387242 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:12.386830    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht" event={"ID":"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea","Type":"ContainerStarted","Data":"5c8a17a2e4a3416b9a1df395838abd696c8a4ae16c6c7719075a59eb32e9baaf"}
Oct 30 20:36:12.505214 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:12.505159    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht" podStartSLOduration=1.505145604 podStartE2EDuration="1.505145604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:12.503442909 +0000 UTC m=+992.182071536" watchObservedRunningTime="2026-10-30 20:36:12.505145604 +0000 UTC m=+992.183774217"
Oct 30 20:36:12.659098 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:12.659011    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:21.659803 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:21.659762    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:21.672873 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:21.672841    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:22.099933 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.099852    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"]
Oct 30 20:36:22.103405 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.103383    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.154822 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.154784    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-4ee72d-openshift-default-dockercfg-5dldm\""
Oct 30 20:36:22.211821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.211784    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"]
Oct 30 20:36:22.277262 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.277223    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.277262 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.277260    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f068d45a-f572-4549-867a-7073169b5f05-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.277552 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.277294    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.277552 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.277461    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.277552 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.277510    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxpps\" (UniqueName: \"kubernetes.io/projected/f068d45a-f572-4549-867a-7073169b5f05-kube-api-access-rxpps\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.277675 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.277587    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f068d45a-f572-4549-867a-7073169b5f05-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.277675 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.277637    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.277773 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.277727    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f068d45a-f572-4549-867a-7073169b5f05-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.277825 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.277791    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.378888 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.378848    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.378888 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.378889    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f068d45a-f572-4549-867a-7073169b5f05-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.379116 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.378928    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.379116 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.378952    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.379116 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.378974    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxpps\" (UniqueName: \"kubernetes.io/projected/f068d45a-f572-4549-867a-7073169b5f05-kube-api-access-rxpps\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.379116 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.379003    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f068d45a-f572-4549-867a-7073169b5f05-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.379116 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.379043    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.379116 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.379075    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f068d45a-f572-4549-867a-7073169b5f05-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.379116 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.379112    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.379472 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.379316    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.379472 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.379368    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.379574 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.379553    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.379645 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.379627    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.379986 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.379964    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f068d45a-f572-4549-867a-7073169b5f05-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.381497 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.381466    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.381796 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.381778    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f068d45a-f572-4549-867a-7073169b5f05-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.435478 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.435449    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f068d45a-f572-4549-867a-7073169b5f05-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.435633 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.435612    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxpps\" (UniqueName: \"kubernetes.io/projected/f068d45a-f572-4549-867a-7073169b5f05-kube-api-access-rxpps\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:22.444796 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.444768    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:36:22.717008 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:22.716910    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:23.144512 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:23.144483    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"]
Oct 30 20:36:23.146805 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:36:23.146779    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf068d45a_f572_4549_867a_7073169b5f05.slice/crio-b760b8a2f9bf2434d2b93661b1bd807e1f407d07f85e9fb1b1ebed2e73a19aa1 WatchSource:0}: Error finding container b760b8a2f9bf2434d2b93661b1bd807e1f407d07f85e9fb1b1ebed2e73a19aa1: Status 404 returned error can't find the container with id b760b8a2f9bf2434d2b93661b1bd807e1f407d07f85e9fb1b1ebed2e73a19aa1
Oct 30 20:36:23.149053 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:23.149018    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:36:23.149121 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:23.149093    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:36:23.149162 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:23.149122    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:36:23.441015 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:23.440910    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw" event={"ID":"f068d45a-f572-4549-867a-7073169b5f05","Type":"ContainerStarted","Data":"52ca8029d60fd80360751f63d7f752f43f809a18d9f111f14db9df8c27abbacf"}
Oct 30 20:36:23.441015 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:23.440979    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw" event={"ID":"f068d45a-f572-4549-867a-7073169b5f05","Type":"ContainerStarted","Data":"b760b8a2f9bf2434d2b93661b1bd807e1f407d07f85e9fb1b1ebed2e73a19aa1"}
Oct 30 20:36:23.560065 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:23.560008    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw" podStartSLOduration=1.559990414 podStartE2EDuration="1.559990414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:23.557759742 +0000 UTC m=+1003.236388357" watchObservedRunningTime="2026-10-30 20:36:23.559990414 +0000 UTC m=+1003.238619028"
Oct 30 20:36:23.717736 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:23.717646    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:24.719447 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:24.719412    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 20:36:24.719859 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:24.719470    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw" podUID="f068d45a-f572-4549-867a-7073169b5f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 20:36:25.717946 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:25.717916    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 20:36:25.718135 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:25.717966    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw" podUID="f068d45a-f572-4549-867a-7073169b5f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 20:36:25.793880 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:25.793844    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"]
Oct 30 20:36:25.797362 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:25.797343    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:25.895217 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:25.895171    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"]
Oct 30 20:36:25.915270 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:25.915218    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:25.915531 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:25.915290    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:25.915531 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:25.915382    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:25.915531 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:25.915411    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:25.915531 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:25.915482    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:25.915759 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:25.915588    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:25.915759 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:25.915628    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cb4f\" (UniqueName: \"kubernetes.io/projected/69467a1c-7315-4154-85a4-72473a1ecd9d-kube-api-access-9cb4f\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:25.915759 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:25.915696    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:25.915759 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:25.915717    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69467a1c-7315-4154-85a4-72473a1ecd9d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:26.017178 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.017076    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:26.017178 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.017140    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:26.017178 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.017176    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:26.017544 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.017208    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:26.017544 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.017251    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:26.017544 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.017304    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cb4f\" (UniqueName: \"kubernetes.io/projected/69467a1c-7315-4154-85a4-72473a1ecd9d-kube-api-access-9cb4f\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:26.017544 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.017381    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:26.017544 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.017414    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69467a1c-7315-4154-85a4-72473a1ecd9d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:26.017544 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.017463    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:26.017843 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.017698    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:26.017843 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.017711    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:26.017843 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.017818    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:26.018154 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.017950    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:26.018271 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.018251    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69467a1c-7315-4154-85a4-72473a1ecd9d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:26.019638 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.019620    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:26.019877 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.019859    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:26.074280 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.074228    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:26.074280 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.074279    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cb4f\" (UniqueName: \"kubernetes.io/projected/69467a1c-7315-4154-85a4-72473a1ecd9d-kube-api-access-9cb4f\") pod \"maas-default-gateway-openshift-default-6886b96f9-ctppg\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:26.110421 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.110383    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:26.569824 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.569786    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"]
Oct 30 20:36:26.570221 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:36:26.570198    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69467a1c_7315_4154_85a4_72473a1ecd9d.slice/crio-a0d22adf94f228263cde559ee64a7113fbc39b181eecc3d8d2f323552451ff03 WatchSource:0}: Error finding container a0d22adf94f228263cde559ee64a7113fbc39b181eecc3d8d2f323552451ff03: Status 404 returned error can't find the container with id a0d22adf94f228263cde559ee64a7113fbc39b181eecc3d8d2f323552451ff03
Oct 30 20:36:26.572636 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.572599    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:36:26.572740 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.572679    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:36:26.572740 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.572722    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:36:26.717511 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.717470    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 20:36:26.717697 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:26.717540    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw" podUID="f068d45a-f572-4549-867a-7073169b5f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 20:36:27.497583 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:27.497534    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg" event={"ID":"69467a1c-7315-4154-85a4-72473a1ecd9d","Type":"ContainerStarted","Data":"87c6e80a103f5b3b971b5012e650c4d518458f0f13874bb342bfe470317df0bb"}
Oct 30 20:36:27.498087 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:27.497604    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg" event={"ID":"69467a1c-7315-4154-85a4-72473a1ecd9d","Type":"ContainerStarted","Data":"a0d22adf94f228263cde559ee64a7113fbc39b181eecc3d8d2f323552451ff03"}
Oct 30 20:36:27.717931 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:27.717889    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 20:36:27.718098 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:27.717955    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw" podUID="f068d45a-f572-4549-867a-7073169b5f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 20:36:27.723877 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:27.723826    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg" podStartSLOduration=2.7238106220000002 podStartE2EDuration="2.723810622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:27.673850078 +0000 UTC m=+1007.352478694" watchObservedRunningTime="2026-10-30 20:36:27.723810622 +0000 UTC m=+1007.402439238"
Oct 30 20:36:28.110712 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:28.110684    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:28.112141 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:28.112113    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ctppg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 20:36:28.112258 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:28.112156    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg" podUID="69467a1c-7315-4154-85a4-72473a1ecd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 20:36:28.717664 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:28.717632    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 20:36:28.718058 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:28.717683    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw" podUID="f068d45a-f572-4549-867a-7073169b5f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 20:36:29.111858 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:29.111769    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ctppg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 20:36:29.111858 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:29.111829    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg" podUID="69467a1c-7315-4154-85a4-72473a1ecd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 20:36:29.718149 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:29.718109    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 20:36:29.718671 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:29.718185    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw" podUID="f068d45a-f572-4549-867a-7073169b5f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 20:36:30.111032 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:30.110940    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ctppg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 20:36:30.111032 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:30.110993    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg" podUID="69467a1c-7315-4154-85a4-72473a1ecd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 20:36:30.722309 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:30.722278    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:30.722814 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:30.722726    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:30.723524 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:30.723492    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 20:36:30.975460 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:36:30.975375    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:36:31.110751 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:31.110718    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ctppg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 20:36:31.110914 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:31.110788    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg" podUID="69467a1c-7315-4154-85a4-72473a1ecd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 20:36:32.111877 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:32.111839    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ctppg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 20:36:32.112300 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:32.111906    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg" podUID="69467a1c-7315-4154-85a4-72473a1ecd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 20:36:33.111674 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:33.111638    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-ctppg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 30 20:36:33.111897 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:33.111707    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg" podUID="69467a1c-7315-4154-85a4-72473a1ecd9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 30 20:36:34.882091 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:34.882057    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:34.882594 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:34.882361    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:34.883358 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:34.883336    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:36:37.805348 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.805290    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"]
Oct 30 20:36:37.817173 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.817145    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.870524 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.870447    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"]
Oct 30 20:36:37.878682 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.878648    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3613a886-6b08-4e25-807a-fbbe5c080775-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.878833 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.878704    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.878833 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.878727    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.878833 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.878813    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3613a886-6b08-4e25-807a-fbbe5c080775-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.878974 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.878946    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.879018 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.879004    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3613a886-6b08-4e25-807a-fbbe5c080775-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.879051 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.879029    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5lrg\" (UniqueName: \"kubernetes.io/projected/3613a886-6b08-4e25-807a-fbbe5c080775-kube-api-access-x5lrg\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.879081 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.879058    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.879113 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.879101    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.917415 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.917385    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ae2d38-openshift-default-dockercfg-9z4cf\""
Oct 30 20:36:37.980692 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.980654    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3613a886-6b08-4e25-807a-fbbe5c080775-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.980910 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.980712    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.980910 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.980736    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.980910 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.980761    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3613a886-6b08-4e25-807a-fbbe5c080775-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.980910 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.980822    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.980910 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.980876    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3613a886-6b08-4e25-807a-fbbe5c080775-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.980910 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.980905    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5lrg\" (UniqueName: \"kubernetes.io/projected/3613a886-6b08-4e25-807a-fbbe5c080775-kube-api-access-x5lrg\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.981451 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.981427    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.981770 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.981462    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.981770 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.981715    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3613a886-6b08-4e25-807a-fbbe5c080775-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.981894 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.981845    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.981974 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.981951    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.982285 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.982263    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.982392 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.982356    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.983872 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.983838    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3613a886-6b08-4e25-807a-fbbe5c080775-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:37.984371 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:37.984352    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:38.039907 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:38.039877    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3613a886-6b08-4e25-807a-fbbe5c080775-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:38.040585 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:38.040562    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5lrg\" (UniqueName: \"kubernetes.io/projected/3613a886-6b08-4e25-807a-fbbe5c080775-kube-api-access-x5lrg\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:38.130571 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:38.130543    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:38.568502 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:38.568418    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"]
Oct 30 20:36:38.597016 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:36:38.596987    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3613a886_6b08_4e25_807a_fbbe5c080775.slice/crio-dc0b7a23bba98ac24bbfccfce031a8e810d75bf249858964c2d723ac7c6d5cb7 WatchSource:0}: Error finding container dc0b7a23bba98ac24bbfccfce031a8e810d75bf249858964c2d723ac7c6d5cb7: Status 404 returned error can't find the container with id dc0b7a23bba98ac24bbfccfce031a8e810d75bf249858964c2d723ac7c6d5cb7
Oct 30 20:36:38.599353 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:38.599300    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:36:38.599420 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:38.599389    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:36:38.599453 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:38.599419    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:36:39.570930 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:39.570880    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c" event={"ID":"3613a886-6b08-4e25-807a-fbbe5c080775","Type":"ContainerStarted","Data":"901cb1872378aa2a25d3b86cff3a747984eece3970a3ce8fce5c2af913b6395a"}
Oct 30 20:36:39.571416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:39.570948    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c" event={"ID":"3613a886-6b08-4e25-807a-fbbe5c080775","Type":"ContainerStarted","Data":"dc0b7a23bba98ac24bbfccfce031a8e810d75bf249858964c2d723ac7c6d5cb7"}
Oct 30 20:36:39.740643 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:39.740586    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c" podStartSLOduration=2.740569836 podStartE2EDuration="2.740569836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:39.738533559 +0000 UTC m=+1019.417162185" watchObservedRunningTime="2026-10-30 20:36:39.740569836 +0000 UTC m=+1019.419198450"
Oct 30 20:36:40.131017 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:40.130984    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:40.132594 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:40.132574    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 20:36:40.132656 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:40.132627    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c" podUID="3613a886-6b08-4e25-807a-fbbe5c080775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 20:36:41.133343 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:41.133137    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 20:36:41.133343 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:41.133197    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c" podUID="3613a886-6b08-4e25-807a-fbbe5c080775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 20:36:42.131648 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:42.131616    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 20:36:42.131844 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:42.131701    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c" podUID="3613a886-6b08-4e25-807a-fbbe5c080775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 20:36:43.131175 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:43.131135    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 20:36:43.131597 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:43.131207    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c" podUID="3613a886-6b08-4e25-807a-fbbe5c080775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 20:36:44.131435 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:44.131398    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 20:36:44.131917 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:44.131459    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c" podUID="3613a886-6b08-4e25-807a-fbbe5c080775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 20:36:45.131114 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:45.131078    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 20:36:45.131277 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:45.131129    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c" podUID="3613a886-6b08-4e25-807a-fbbe5c080775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 20:36:46.192556 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:46.192514    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:46.192958 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:46.192914    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:36:46.193544 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:36:46.193529    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:37:00.955343 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:37:00.955244    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:37:06.623645 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:06.623610    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"]
Oct 30 20:37:06.627310 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:06.627289    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:06.729418 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:06.729386    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ae2d38\"/\"distinct-w3-ae2d38-kserve-self-signed-certs\""
Oct 30 20:37:06.731293 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:06.731264    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"]
Oct 30 20:37:06.799471 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:06.798821    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-tmp-dir\") pod \"distinct-w3-ae2d38-kserve-84564db7fc-979bf\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:06.799471 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:06.799032    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf89b9b8-0419-48bc-b410-12807dab75db-tls-certs\") pod \"distinct-w3-ae2d38-kserve-84564db7fc-979bf\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:06.799471 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:06.799139    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cznfl\" (UniqueName: \"kubernetes.io/projected/bf89b9b8-0419-48bc-b410-12807dab75db-kube-api-access-cznfl\") pod \"distinct-w3-ae2d38-kserve-84564db7fc-979bf\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:06.799471 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:06.799169    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-dshm\") pod \"distinct-w3-ae2d38-kserve-84564db7fc-979bf\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:06.799471 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:06.799216    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-home\") pod \"distinct-w3-ae2d38-kserve-84564db7fc-979bf\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:06.799471 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:06.799280    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-model-cache\") pod \"distinct-w3-ae2d38-kserve-84564db7fc-979bf\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:06.900680 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:06.900647    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf89b9b8-0419-48bc-b410-12807dab75db-tls-certs\") pod \"distinct-w3-ae2d38-kserve-84564db7fc-979bf\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:06.900909 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:06.900719    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cznfl\" (UniqueName: \"kubernetes.io/projected/bf89b9b8-0419-48bc-b410-12807dab75db-kube-api-access-cznfl\") pod \"distinct-w3-ae2d38-kserve-84564db7fc-979bf\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:06.900909 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:06.900749    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-dshm\") pod \"distinct-w3-ae2d38-kserve-84564db7fc-979bf\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:06.900909 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:06.900802    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-home\") pod \"distinct-w3-ae2d38-kserve-84564db7fc-979bf\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:06.900909 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:06.900854    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-model-cache\") pod \"distinct-w3-ae2d38-kserve-84564db7fc-979bf\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:06.901107 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:06.900916    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-tmp-dir\") pod \"distinct-w3-ae2d38-kserve-84564db7fc-979bf\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:06.901586 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:06.901551    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-home\") pod \"distinct-w3-ae2d38-kserve-84564db7fc-979bf\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:06.901736 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:06.901589    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-model-cache\") pod \"distinct-w3-ae2d38-kserve-84564db7fc-979bf\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:06.901801 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:06.901597    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-tmp-dir\") pod \"distinct-w3-ae2d38-kserve-84564db7fc-979bf\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:06.903574 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:06.903541    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-dshm\") pod \"distinct-w3-ae2d38-kserve-84564db7fc-979bf\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:06.903696 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:06.903643    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf89b9b8-0419-48bc-b410-12807dab75db-tls-certs\") pod \"distinct-w3-ae2d38-kserve-84564db7fc-979bf\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:06.957923 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:06.957881    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cznfl\" (UniqueName: \"kubernetes.io/projected/bf89b9b8-0419-48bc-b410-12807dab75db-kube-api-access-cznfl\") pod \"distinct-w3-ae2d38-kserve-84564db7fc-979bf\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:07.240744 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:07.240598    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:07.622918 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:07.622891    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"]
Oct 30 20:37:07.625430 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:37:07.625391    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf89b9b8_0419_48bc_b410_12807dab75db.slice/crio-fda68d39ee39e0044e48921699ddd611d53dec2cf6c496179f5e6e8e76eeff4a WatchSource:0}: Error finding container fda68d39ee39e0044e48921699ddd611d53dec2cf6c496179f5e6e8e76eeff4a: Status 404 returned error can't find the container with id fda68d39ee39e0044e48921699ddd611d53dec2cf6c496179f5e6e8e76eeff4a
Oct 30 20:37:07.713573 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:07.713533    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf" event={"ID":"bf89b9b8-0419-48bc-b410-12807dab75db","Type":"ContainerStarted","Data":"864dd860845797f85fd55c5d481611ef736050a92557acb65fc07b87404709ad"}
Oct 30 20:37:07.713748 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:07.713584    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf" event={"ID":"bf89b9b8-0419-48bc-b410-12807dab75db","Type":"ContainerStarted","Data":"fda68d39ee39e0044e48921699ddd611d53dec2cf6c496179f5e6e8e76eeff4a"}
Oct 30 20:37:07.830208 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:07.830149    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf" podStartSLOduration=1.8301332019999998 podStartE2EDuration="1.830133202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:07.829413437 +0000 UTC m=+1047.508042056" watchObservedRunningTime="2026-10-30 20:37:07.830133202 +0000 UTC m=+1047.508761816"
Oct 30 20:37:08.241604 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:08.241563    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:17.241273 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:17.241237    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:17.253995 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:17.253967    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:17.784716 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:37:17.784684    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:37:30.953223 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:37:30.953190    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:38:00.953966 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:38:00.953930    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:38:08.571875 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.571835    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"]
Oct 30 20:38:08.575555 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.575531    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.627556 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.627515    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-workload-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.627750 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.627569    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-credential-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.627750 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.627635    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-data\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.627750 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.627681    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa37152b-e260-4cd0-9c00-528cc696aaae-istiod-ca-cert\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.627750 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.627753    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvnt2\" (UniqueName: \"kubernetes.io/projected/fa37152b-e260-4cd0-9c00-528cc696aaae-kube-api-access-mvnt2\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.627976 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.627801    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-podinfo\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.627976 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.627803    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5f9b95-openshift-default-dockercfg-kpgjw\""
Oct 30 20:38:08.627976 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.627819    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-token\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.627976 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.627838    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-envoy\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.628128 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.627977    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-workload-certs\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.679975 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.679948    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"]
Oct 30 20:38:08.729390 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.729359    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-workload-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.729567 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.729401    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-credential-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.729567 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.729419    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-data\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.729567 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.729444    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa37152b-e260-4cd0-9c00-528cc696aaae-istiod-ca-cert\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.729567 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.729492    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvnt2\" (UniqueName: \"kubernetes.io/projected/fa37152b-e260-4cd0-9c00-528cc696aaae-kube-api-access-mvnt2\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.729567 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.729522    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-podinfo\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.729567 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.729547    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-token\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.729881 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.729577    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-envoy\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.729881 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.729664    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-workload-certs\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.729881 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.729793    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-workload-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.729881 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.729847    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-data\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.730112 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.730021    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-workload-certs\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.730161 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.730113    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-credential-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.730203 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.730165    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa37152b-e260-4cd0-9c00-528cc696aaae-istiod-ca-cert\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.731916 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.731896    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-envoy\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.732102 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.732082    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-podinfo\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.785827 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.785792    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-token\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.786116 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.786098    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvnt2\" (UniqueName: \"kubernetes.io/projected/fa37152b-e260-4cd0-9c00-528cc696aaae-kube-api-access-mvnt2\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:08.887808 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:08.887774    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:09.313063 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:09.313034    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"]
Oct 30 20:38:09.315480 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:38:09.315452    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa37152b_e260_4cd0_9c00_528cc696aaae.slice/crio-eae935b02a3e29ffb9bd0831bd8a902bfde7907fa846182da372f22b1837f2f7 WatchSource:0}: Error finding container eae935b02a3e29ffb9bd0831bd8a902bfde7907fa846182da372f22b1837f2f7: Status 404 returned error can't find the container with id eae935b02a3e29ffb9bd0831bd8a902bfde7907fa846182da372f22b1837f2f7
Oct 30 20:38:09.317808 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:09.317773    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:38:09.317901 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:09.317841    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:38:09.317901 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:09.317868    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:38:10.026373 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:10.026338    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978" event={"ID":"fa37152b-e260-4cd0-9c00-528cc696aaae","Type":"ContainerStarted","Data":"5da9e61983809800b300a6113e15c4e43297e3986c922dd1e9f497cffb3dc260"}
Oct 30 20:38:10.026830 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:10.026391    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978" event={"ID":"fa37152b-e260-4cd0-9c00-528cc696aaae","Type":"ContainerStarted","Data":"eae935b02a3e29ffb9bd0831bd8a902bfde7907fa846182da372f22b1837f2f7"}
Oct 30 20:38:10.162844 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:10.162782    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978" podStartSLOduration=2.162767445 podStartE2EDuration="2.162767445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:10.160803344 +0000 UTC m=+1109.839431959" watchObservedRunningTime="2026-10-30 20:38:10.162767445 +0000 UTC m=+1109.841396060"
Oct 30 20:38:10.888642 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:10.888603    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:10.890089 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:10.890064    4471 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 20:38:10.890215 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:10.890111    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978" podUID="fa37152b-e260-4cd0-9c00-528cc696aaae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 20:38:11.889129 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:11.889098    4471 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 20:38:11.889619 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:11.889149    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978" podUID="fa37152b-e260-4cd0-9c00-528cc696aaae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 20:38:12.888781 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:12.888739    4471 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 20:38:12.888954 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:12.888811    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978" podUID="fa37152b-e260-4cd0-9c00-528cc696aaae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 20:38:13.888494 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:13.888465    4471 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 20:38:13.888918 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:13.888518    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978" podUID="fa37152b-e260-4cd0-9c00-528cc696aaae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 20:38:14.889099 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:14.889057    4471 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 20:38:14.889510 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:14.889126    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978" podUID="fa37152b-e260-4cd0-9c00-528cc696aaae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 20:38:15.888466 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:15.888424    4471 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 20:38:15.888678 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:15.888479    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978" podUID="fa37152b-e260-4cd0-9c00-528cc696aaae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 20:38:16.643980 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.643941    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"]
Oct 30 20:38:16.647843 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.647826    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:16.728498 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.728439    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfg5q\" (UniqueName: \"kubernetes.io/projected/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-kube-api-access-hfg5q\") pod \"distinct-2-w0-0a78ac-kserve-558b9b874-78zdk\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:16.728709 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.728562    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-home\") pod \"distinct-2-w0-0a78ac-kserve-558b9b874-78zdk\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:16.728709 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.728597    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-dshm\") pod \"distinct-2-w0-0a78ac-kserve-558b9b874-78zdk\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:16.728709 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.728644    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-tls-certs\") pod \"distinct-2-w0-0a78ac-kserve-558b9b874-78zdk\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:16.728879 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.728776    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-model-cache\") pod \"distinct-2-w0-0a78ac-kserve-558b9b874-78zdk\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:16.728879 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.728806    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-tmp-dir\") pod \"distinct-2-w0-0a78ac-kserve-558b9b874-78zdk\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:16.749996 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.749963    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0a78ac\"/\"distinct-2-w0-0a78ac-kserve-self-signed-certs\""
Oct 30 20:38:16.752211 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.752188    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"]
Oct 30 20:38:16.829929 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.829891    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfg5q\" (UniqueName: \"kubernetes.io/projected/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-kube-api-access-hfg5q\") pod \"distinct-2-w0-0a78ac-kserve-558b9b874-78zdk\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:16.830119 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.829948    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-home\") pod \"distinct-2-w0-0a78ac-kserve-558b9b874-78zdk\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:16.830119 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.830032    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-dshm\") pod \"distinct-2-w0-0a78ac-kserve-558b9b874-78zdk\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:16.830119 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.830112    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-tls-certs\") pod \"distinct-2-w0-0a78ac-kserve-558b9b874-78zdk\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:16.830307 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.830156    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-model-cache\") pod \"distinct-2-w0-0a78ac-kserve-558b9b874-78zdk\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:16.830307 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.830289    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-home\") pod \"distinct-2-w0-0a78ac-kserve-558b9b874-78zdk\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:16.830449 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.830305    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-tmp-dir\") pod \"distinct-2-w0-0a78ac-kserve-558b9b874-78zdk\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:16.830525 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.830504    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-model-cache\") pod \"distinct-2-w0-0a78ac-kserve-558b9b874-78zdk\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:16.830567 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.830521    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-tmp-dir\") pod \"distinct-2-w0-0a78ac-kserve-558b9b874-78zdk\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:16.832504 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.832480    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-dshm\") pod \"distinct-2-w0-0a78ac-kserve-558b9b874-78zdk\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:16.832666 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.832651    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-tls-certs\") pod \"distinct-2-w0-0a78ac-kserve-558b9b874-78zdk\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:16.887335 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.887283    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfg5q\" (UniqueName: \"kubernetes.io/projected/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-kube-api-access-hfg5q\") pod \"distinct-2-w0-0a78ac-kserve-558b9b874-78zdk\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") " pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:16.892639 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.892612    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:16.893008 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.892990    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:16.893730 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.893710    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:38:16.959545 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:16.959464    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:17.398359 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:17.398313    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"]
Oct 30 20:38:17.400114 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:38:17.400085    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0e9e4ae_4ffa_43ee_bce5_26b21c4c3f98.slice/crio-ffa66152675dc7d3a3a9bbc80c069b6c03df71de3e51d34531c165b0eb3bdda9 WatchSource:0}: Error finding container ffa66152675dc7d3a3a9bbc80c069b6c03df71de3e51d34531c165b0eb3bdda9: Status 404 returned error can't find the container with id ffa66152675dc7d3a3a9bbc80c069b6c03df71de3e51d34531c165b0eb3bdda9
Oct 30 20:38:18.104510 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:18.104467    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk" event={"ID":"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98","Type":"ContainerStarted","Data":"7feb2c5b2fa335e2980d845e6cb99260508f72fbf9d10653dc974917ae5accc1"}
Oct 30 20:38:18.104921 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:18.104521    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk" event={"ID":"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98","Type":"ContainerStarted","Data":"ffa66152675dc7d3a3a9bbc80c069b6c03df71de3e51d34531c165b0eb3bdda9"}
Oct 30 20:38:18.220361 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:18.220287    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk" podStartSLOduration=2.22027443 podStartE2EDuration="2.22027443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:18.218467173 +0000 UTC m=+1117.897095783" watchObservedRunningTime="2026-10-30 20:38:18.22027443 +0000 UTC m=+1117.898903044"
Oct 30 20:38:18.960283 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:18.960249    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:26.960082 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:26.959994    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:26.972511 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:26.972483    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:27.159124 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:38:27.159091    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:38:30.953293 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:38:30.953258    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:39:00.955207 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:39:00.955162    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:39:19.542538 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.542504    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"]
Oct 30 20:39:19.545703 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.545675    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:19.597579 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.597559    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ae2d38\"/\"distinct-2-w3-ae2d38-kserve-self-signed-certs\""
Oct 30 20:39:19.606511 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.606483    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"]
Oct 30 20:39:19.675316 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.675290    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-dshm\") pod \"distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:19.675440 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.675347    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-home\") pod \"distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:19.675440 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.675366    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-model-cache\") pod \"distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:19.675440 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.675407    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4344406-c869-420f-8cf3-93c5e0549be8-tls-certs\") pod \"distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:19.675536 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.675509    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4glb\" (UniqueName: \"kubernetes.io/projected/a4344406-c869-420f-8cf3-93c5e0549be8-kube-api-access-g4glb\") pod \"distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:19.675573 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.675541    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-tmp-dir\") pod \"distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:19.776777 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.776744    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-dshm\") pod \"distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:19.776973 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.776795    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-home\") pod \"distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:19.776973 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.776907    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-model-cache\") pod \"distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:19.776973 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.776934    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4344406-c869-420f-8cf3-93c5e0549be8-tls-certs\") pod \"distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:19.777137 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.776985    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4glb\" (UniqueName: \"kubernetes.io/projected/a4344406-c869-420f-8cf3-93c5e0549be8-kube-api-access-g4glb\") pod \"distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:19.777137 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.777025    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-tmp-dir\") pod \"distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:19.777271 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.777249    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-home\") pod \"distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:19.777674 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.777627    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-tmp-dir\") pod \"distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:19.777796 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.777696    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-model-cache\") pod \"distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:19.779204 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.779151    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-dshm\") pod \"distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:19.779666 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.779642    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4344406-c869-420f-8cf3-93c5e0549be8-tls-certs\") pod \"distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:19.833870 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.833810    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4glb\" (UniqueName: \"kubernetes.io/projected/a4344406-c869-420f-8cf3-93c5e0549be8-kube-api-access-g4glb\") pod \"distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") " pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:19.856788 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:19.856760    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:20.293458 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:20.293434    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"]
Oct 30 20:39:20.295253 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:39:20.295226    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4344406_c869_420f_8cf3_93c5e0549be8.slice/crio-b7f899b94d9be5d752907f9038bcc16a326d38f763cab308f1b1b533a85cd0dd WatchSource:0}: Error finding container b7f899b94d9be5d752907f9038bcc16a326d38f763cab308f1b1b533a85cd0dd: Status 404 returned error can't find the container with id b7f899b94d9be5d752907f9038bcc16a326d38f763cab308f1b1b533a85cd0dd
Oct 30 20:39:20.401863 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:20.401829    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl" event={"ID":"a4344406-c869-420f-8cf3-93c5e0549be8","Type":"ContainerStarted","Data":"778b52228592ae82ecfc42f9b9fe3498024af28f8f7f9b397167c5fc40634657"}
Oct 30 20:39:20.402018 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:20.401878    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl" event={"ID":"a4344406-c869-420f-8cf3-93c5e0549be8","Type":"ContainerStarted","Data":"b7f899b94d9be5d752907f9038bcc16a326d38f763cab308f1b1b533a85cd0dd"}
Oct 30 20:39:20.520430 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:20.520378    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl" podStartSLOduration=1.520363414 podStartE2EDuration="1.520363414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:20.51713692 +0000 UTC m=+1180.195765533" watchObservedRunningTime="2026-10-30 20:39:20.520363414 +0000 UTC m=+1180.198992028"
Oct 30 20:39:20.857121 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:20.857090    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:29.857455 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:29.857420    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:29.869410 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:29.869386    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:30.467507 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:39:30.467482    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:39:30.953023 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:39:30.952989    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:40:00.953606 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:40:00.953535    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:40:09.418957 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.418923    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"]
Oct 30 20:40:09.422504 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.422487    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.483843 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.483815    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"]
Oct 30 20:40:09.610770 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.610743    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.610923 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.610791    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.610923 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.610879    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49dws\" (UniqueName: \"kubernetes.io/projected/7b68692e-aa7c-4b66-a316-19e71881d64e-kube-api-access-49dws\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.611001 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.610971    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.611045 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.611029    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.611084 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.611060    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.611127 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.611088    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.611205 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.611181    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b68692e-aa7c-4b66-a316-19e71881d64e-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.611260 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.611238    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.712386 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.712265    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.712386 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.712310    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b68692e-aa7c-4b66-a316-19e71881d64e-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.712386 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.712366    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.712673 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.712416    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.712673 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.712461    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.712673 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.712522    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49dws\" (UniqueName: \"kubernetes.io/projected/7b68692e-aa7c-4b66-a316-19e71881d64e-kube-api-access-49dws\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.712673 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.712552    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.712673 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.712631    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.712673 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.712657    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.712972 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.712776    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.713031 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.712976    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.713085 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.713039    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.713175 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.713157    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.714408 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.714381    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b68692e-aa7c-4b66-a316-19e71881d64e-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.715059 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.715031    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.715168 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.715094    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.768248 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.768219    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:09.768395 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:09.768264    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49dws\" (UniqueName: \"kubernetes.io/projected/7b68692e-aa7c-4b66-a316-19e71881d64e-kube-api-access-49dws\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:10.035920 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:10.035853    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:10.463738 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:10.463716    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"]
Oct 30 20:40:10.465311 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:40:10.465280    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b68692e_aa7c_4b66_a316_19e71881d64e.slice/crio-24ab28322e8be1c38d00802f70d50303281e9caf349b6e9f4aa09153e7fd532d WatchSource:0}: Error finding container 24ab28322e8be1c38d00802f70d50303281e9caf349b6e9f4aa09153e7fd532d: Status 404 returned error can't find the container with id 24ab28322e8be1c38d00802f70d50303281e9caf349b6e9f4aa09153e7fd532d
Oct 30 20:40:10.467485 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:10.467456    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:40:10.467562 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:10.467534    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:40:10.467602 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:10.467561    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:40:10.687350 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:10.686302    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw" event={"ID":"7b68692e-aa7c-4b66-a316-19e71881d64e","Type":"ContainerStarted","Data":"e58f8fcb15afa6e447fcdbb5bfe44e25693cdca99d1c1a953090e15fe27740cb"}
Oct 30 20:40:10.687350 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:10.686395    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw" event={"ID":"7b68692e-aa7c-4b66-a316-19e71881d64e","Type":"ContainerStarted","Data":"24ab28322e8be1c38d00802f70d50303281e9caf349b6e9f4aa09153e7fd532d"}
Oct 30 20:40:10.817099 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:10.816994    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw" podStartSLOduration=1.816976 podStartE2EDuration="1.816976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:10.815633308 +0000 UTC m=+1230.494261922" watchObservedRunningTime="2026-10-30 20:40:10.816976 +0000 UTC m=+1230.495604617"
Oct 30 20:40:11.036741 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:11.036708    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:11.038205 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:11.038174    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 20:40:11.038402 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:11.038228    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw" podUID="7b68692e-aa7c-4b66-a316-19e71881d64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 20:40:12.037207 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:12.037172    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 20:40:12.037752 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:12.037225    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw" podUID="7b68692e-aa7c-4b66-a316-19e71881d64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 20:40:13.036306 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:13.036274    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 20:40:13.036498 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:13.036346    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw" podUID="7b68692e-aa7c-4b66-a316-19e71881d64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 20:40:14.036571 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:14.036534    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 20:40:14.036950 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:14.036594    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw" podUID="7b68692e-aa7c-4b66-a316-19e71881d64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 20:40:15.036517 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:15.036481    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 20:40:15.036693 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:15.036534    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw" podUID="7b68692e-aa7c-4b66-a316-19e71881d64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 20:40:16.036873 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:16.036841    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 20:40:16.037265 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:16.036897    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw" podUID="7b68692e-aa7c-4b66-a316-19e71881d64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 20:40:17.036822 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:17.036790    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 20:40:17.037008 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:17.036844    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw" podUID="7b68692e-aa7c-4b66-a316-19e71881d64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 20:40:18.041145 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:18.041109    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:18.041847 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:18.041814    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:18.042376 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:18.042357    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 20:40:30.954125 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:40:30.954092    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:40:37.479338 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.479278    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"]
Oct 30 20:40:37.482920 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.482899    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.528929 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.528894    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"]
Oct 30 20:40:37.532570 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.532551    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.540328 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.540300    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0a78ac-openshift-default-dockercfg-6pwsb\""
Oct 30 20:40:37.617759 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.617730    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.617759 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.617761    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.617965 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.617791    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.617965 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.617922    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.617965 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.617957    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.618124 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.617994    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.618124 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.618076    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.618231 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.618143    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.618231 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.618171    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.618365 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.618262    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.618365 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.618298    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.618365 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.618354    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh24t\" (UniqueName: \"kubernetes.io/projected/19dfa0fc-a01f-495b-8d9f-e6e976bef433-kube-api-access-bh24t\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.618542 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.618373    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.618542 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.618390    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsp2j\" (UniqueName: \"kubernetes.io/projected/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-kube-api-access-wsp2j\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.618542 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.618436    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.618542 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.618481    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.618542 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.618503    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.618542 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.618519    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.681195 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.681167    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"]
Oct 30 20:40:37.694861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.694833    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"]
Oct 30 20:40:37.719210 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.719181    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.719371 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.719223    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.719371 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.719273    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.719371 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.719296    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.719371 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.719354    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.719597 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.719388    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.719597 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.719410    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.719597 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.719443    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.719597 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.719477    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.719597 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.719548    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.719597 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.719577    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.719950 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.719613    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.719950 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.719624    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.719950 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.719643    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.719950 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.719703    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh24t\" (UniqueName: \"kubernetes.io/projected/19dfa0fc-a01f-495b-8d9f-e6e976bef433-kube-api-access-bh24t\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.719950 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.719729    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.719950 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.719734    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.719950 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.719760    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsp2j\" (UniqueName: \"kubernetes.io/projected/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-kube-api-access-wsp2j\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.719950 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.719784    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.719950 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.719819    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.720480 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.720250    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.720654 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.720636    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.720727 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.720635    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.720787 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.720741    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.720928 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.720908    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.721024 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.720998    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.722487 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.722431    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.723380 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.722875    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.723380 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.723287    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.723614 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.723589    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.723815 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.723790    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.725104 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.725081    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.777846 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.777782    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.777937 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.777849    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsp2j\" (UniqueName: \"kubernetes.io/projected/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-kube-api-access-wsp2j\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.777989 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.777761    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.777989 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.777930    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh24t\" (UniqueName: \"kubernetes.io/projected/19dfa0fc-a01f-495b-8d9f-e6e976bef433-kube-api-access-bh24t\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:37.796594 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.796576    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:37.844677 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:37.844647    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:38.226516 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:38.226482    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"]
Oct 30 20:40:38.265586 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:40:38.265548    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35507347_96f8_4fbe_ad4d_49e8ec22eaf7.slice/crio-4d4686bd0fc50aa414d909d4ee7a5d247f42cc3f52752a6ae5e0480aef4c7ca9 WatchSource:0}: Error finding container 4d4686bd0fc50aa414d909d4ee7a5d247f42cc3f52752a6ae5e0480aef4c7ca9: Status 404 returned error can't find the container with id 4d4686bd0fc50aa414d909d4ee7a5d247f42cc3f52752a6ae5e0480aef4c7ca9
Oct 30 20:40:38.267847 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:38.267813    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:40:38.267933 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:38.267878    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:40:38.267933 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:38.267906    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:40:38.281521 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:38.281480    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"]
Oct 30 20:40:38.282681 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:40:38.282662    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19dfa0fc_a01f_495b_8d9f_e6e976bef433.slice/crio-ccaf3baa9d872f52b73e18a8123aa41ef9d9b13ed860b75047ffb1f2e8728b30 WatchSource:0}: Error finding container ccaf3baa9d872f52b73e18a8123aa41ef9d9b13ed860b75047ffb1f2e8728b30: Status 404 returned error can't find the container with id ccaf3baa9d872f52b73e18a8123aa41ef9d9b13ed860b75047ffb1f2e8728b30
Oct 30 20:40:38.285177 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:38.285142    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:40:38.285270 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:38.285227    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:40:38.285341 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:38.285272    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:40:38.844888 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:38.844811    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx" event={"ID":"19dfa0fc-a01f-495b-8d9f-e6e976bef433","Type":"ContainerStarted","Data":"c1896b4b544a8c43b11e19ce75d2cc44a13f9b42de61a5ede3370a5fa29a390e"}
Oct 30 20:40:38.844888 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:38.844919    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx" event={"ID":"19dfa0fc-a01f-495b-8d9f-e6e976bef433","Type":"ContainerStarted","Data":"ccaf3baa9d872f52b73e18a8123aa41ef9d9b13ed860b75047ffb1f2e8728b30"}
Oct 30 20:40:38.846543 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:38.846513    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv" event={"ID":"35507347-96f8-4fbe-ad4d-49e8ec22eaf7","Type":"ContainerStarted","Data":"4b393651c8f5cd7e2a583eea1b82bde6c78923f4a943a67969b646cbc3d5f284"}
Oct 30 20:40:38.846699 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:38.846562    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv" event={"ID":"35507347-96f8-4fbe-ad4d-49e8ec22eaf7","Type":"ContainerStarted","Data":"4d4686bd0fc50aa414d909d4ee7a5d247f42cc3f52752a6ae5e0480aef4c7ca9"}
Oct 30 20:40:38.962978 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:38.962926    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx" podStartSLOduration=1.9629127629999998 podStartE2EDuration="1.962912763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:38.961891903 +0000 UTC m=+1258.640520519" watchObservedRunningTime="2026-10-30 20:40:38.962912763 +0000 UTC m=+1258.641541377"
Oct 30 20:40:39.079950 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:39.079900    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv" podStartSLOduration=2.079885896 podStartE2EDuration="2.079885896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:39.077765936 +0000 UTC m=+1258.756394550" watchObservedRunningTime="2026-10-30 20:40:39.079885896 +0000 UTC m=+1258.758514571"
Oct 30 20:40:39.798348 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:39.797830    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:39.798993 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:39.798966    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 20:40:39.799124 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:39.799096    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv" podUID="35507347-96f8-4fbe-ad4d-49e8ec22eaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 20:40:39.894893 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:39.894505    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:39.896164 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:39.896135    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 20:40:39.896350 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:39.896251    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx" podUID="19dfa0fc-a01f-495b-8d9f-e6e976bef433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 20:40:40.797345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:40.797119    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 20:40:40.797646 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:40.797189    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv" podUID="35507347-96f8-4fbe-ad4d-49e8ec22eaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 20:40:40.862344 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:40.858110    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 20:40:40.862344 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:40.858156    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx" podUID="19dfa0fc-a01f-495b-8d9f-e6e976bef433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 20:40:41.798543 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:41.798496    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 20:40:41.799263 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:41.799231    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv" podUID="35507347-96f8-4fbe-ad4d-49e8ec22eaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 20:40:41.847809 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:41.847776    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 20:40:41.848121 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:41.848076    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx" podUID="19dfa0fc-a01f-495b-8d9f-e6e976bef433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 20:40:42.799362 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:42.797712    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 20:40:42.799362 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:42.797790    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv" podUID="35507347-96f8-4fbe-ad4d-49e8ec22eaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 20:40:42.846667 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:42.846635    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 20:40:42.846892 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:42.846868    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx" podUID="19dfa0fc-a01f-495b-8d9f-e6e976bef433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 20:40:43.797803 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:43.797761    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 20:40:43.798002 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:43.797829    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv" podUID="35507347-96f8-4fbe-ad4d-49e8ec22eaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 20:40:43.866355 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:43.851445    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 20:40:43.866355 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:43.851517    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx" podUID="19dfa0fc-a01f-495b-8d9f-e6e976bef433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 20:40:44.797732 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:44.797692    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 30 20:40:44.797918 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:44.797756    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv" podUID="35507347-96f8-4fbe-ad4d-49e8ec22eaf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 30 20:40:44.845219 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:44.845191    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 20:40:44.845404 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:44.845245    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx" podUID="19dfa0fc-a01f-495b-8d9f-e6e976bef433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 20:40:45.850028 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:45.849998    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:45.850502 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:45.850276    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:45.851432 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:45.851412    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:40:45.949039 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:45.949011    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:45.949922 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:45.949901    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:40:45.950565 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:40:45.950544    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:41:00.954999 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:41:00.954961    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:41:10.039660 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.039615    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"]
Oct 30 20:41:10.044960 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.044937    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.098748 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.098716    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-a78bf4-openshift-default-dockercfg-cn9sv\""
Oct 30 20:41:10.104269 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.104234    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-data\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.104379 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.104281    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-workload-socket\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.104437 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.104402    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-envoy\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.104473 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.104439    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-token\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.104541 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.104519    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istiod-ca-cert\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.104681 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.104660    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2lzr\" (UniqueName: \"kubernetes.io/projected/bba3ab4d-cb5f-45dc-9269-31145e9a396d-kube-api-access-n2lzr\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.104717 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.104694    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-workload-certs\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.104824 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.104808    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-credential-socket\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.104862 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.104833    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-podinfo\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.149777 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.149744    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"]
Oct 30 20:41:10.205958 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.205923    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-credential-socket\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.205958 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.205961    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-podinfo\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.206198 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.205988    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-data\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.206198 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.206115    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-workload-socket\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.206363 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.206210    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-envoy\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.206363 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.206252    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-token\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.206363 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.206316    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istiod-ca-cert\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.206543 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.206420    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-data\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.206543 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.206467    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-credential-socket\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.206543 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.206505    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2lzr\" (UniqueName: \"kubernetes.io/projected/bba3ab4d-cb5f-45dc-9269-31145e9a396d-kube-api-access-n2lzr\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.206712 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.206555    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-workload-certs\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.206712 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.206578    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-workload-socket\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.207041 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.207014    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-workload-certs\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.207152 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.207130    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istiod-ca-cert\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.208971 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.208938    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-envoy\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.209077 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.208996    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-podinfo\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.268397 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.268361    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-token\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.268804 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.268776    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2lzr\" (UniqueName: \"kubernetes.io/projected/bba3ab4d-cb5f-45dc-9269-31145e9a396d-kube-api-access-n2lzr\") pod \"e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") " pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.359152 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.359050    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:10.797942 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.797891    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"]
Oct 30 20:41:10.911130 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.911088    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:41:10.911261 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.911172    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:41:10.911261 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:10.911206    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:41:11.102316 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:11.102215    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m" event={"ID":"bba3ab4d-cb5f-45dc-9269-31145e9a396d","Type":"ContainerStarted","Data":"ff10320d35b9216cfae7f772b91ff37a2db4287679e603303f0cbe0b443eed90"}
Oct 30 20:41:11.102812 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:11.102341    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m" event={"ID":"bba3ab4d-cb5f-45dc-9269-31145e9a396d","Type":"ContainerStarted","Data":"729e9fe28cb14df31b75a74e602439bb494e85c5054bab7b6333dec403714294"}
Oct 30 20:41:11.228474 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:11.228417    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m" podStartSLOduration=2.228400599 podStartE2EDuration="2.228400599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:11.225940753 +0000 UTC m=+1290.904569369" watchObservedRunningTime="2026-10-30 20:41:11.228400599 +0000 UTC m=+1290.907029212"
Oct 30 20:41:11.359410 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:11.359306    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:11.361156 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:11.361122    4471 patch_prober.go:29] interesting pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 20:41:11.361373 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:11.361198    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m" podUID="bba3ab4d-cb5f-45dc-9269-31145e9a396d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 20:41:12.367357 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:12.364811    4471 patch_prober.go:29] interesting pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 20:41:12.367357 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:12.364887    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m" podUID="bba3ab4d-cb5f-45dc-9269-31145e9a396d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 20:41:13.363352 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:13.362806    4471 patch_prober.go:29] interesting pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 20:41:13.363352 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:13.362878    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m" podUID="bba3ab4d-cb5f-45dc-9269-31145e9a396d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 20:41:14.360224 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:14.359748    4471 patch_prober.go:29] interesting pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 20:41:14.360224 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:14.359823    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m" podUID="bba3ab4d-cb5f-45dc-9269-31145e9a396d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 20:41:15.365352 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:15.364820    4471 patch_prober.go:29] interesting pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 20:41:15.365352 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:15.364896    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m" podUID="bba3ab4d-cb5f-45dc-9269-31145e9a396d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 20:41:16.362765 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:16.362721    4471 patch_prober.go:29] interesting pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 20:41:16.363114 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:16.363072    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m" podUID="bba3ab4d-cb5f-45dc-9269-31145e9a396d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 20:41:17.359703 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:17.359661    4471 patch_prober.go:29] interesting pod/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 20:41:17.360153 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:17.359734    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m" podUID="bba3ab4d-cb5f-45dc-9269-31145e9a396d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 20:41:17.506168 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:17.506127    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7269-6b3e9a"]
Oct 30 20:41:17.510180 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:17.510149    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-6b3e9a"
Oct 30 20:41:17.563153 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:17.563022    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vjnsm\""
Oct 30 20:41:17.565860 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:17.565830    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7269-6b3e9a"]
Oct 30 20:41:17.621471 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:17.621378    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq9x5\" (UniqueName: \"kubernetes.io/projected/119e7954-1273-4cf3-853a-11925bd2a4a0-kube-api-access-rq9x5\") pod \"test-curl-7269-6b3e9a\" (UID: \"119e7954-1273-4cf3-853a-11925bd2a4a0\") " pod="opendatahub/test-curl-7269-6b3e9a"
Oct 30 20:41:17.722461 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:17.722418    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq9x5\" (UniqueName: \"kubernetes.io/projected/119e7954-1273-4cf3-853a-11925bd2a4a0-kube-api-access-rq9x5\") pod \"test-curl-7269-6b3e9a\" (UID: \"119e7954-1273-4cf3-853a-11925bd2a4a0\") " pod="opendatahub/test-curl-7269-6b3e9a"
Oct 30 20:41:17.780879 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:17.780841    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq9x5\" (UniqueName: \"kubernetes.io/projected/119e7954-1273-4cf3-853a-11925bd2a4a0-kube-api-access-rq9x5\") pod \"test-curl-7269-6b3e9a\" (UID: \"119e7954-1273-4cf3-853a-11925bd2a4a0\") " pod="opendatahub/test-curl-7269-6b3e9a"
Oct 30 20:41:17.824302 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:17.824252    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-6b3e9a"
Oct 30 20:41:18.218909 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:18.218883    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7269-6b3e9a"]
Oct 30 20:41:18.221240 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:41:18.221210    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod119e7954_1273_4cf3_853a_11925bd2a4a0.slice/crio-ea98116e010d1748481f02ca0bbc0b9b55edcdfd0838a4695cc2e0e705bc0258 WatchSource:0}: Error finding container ea98116e010d1748481f02ca0bbc0b9b55edcdfd0838a4695cc2e0e705bc0258: Status 404 returned error can't find the container with id ea98116e010d1748481f02ca0bbc0b9b55edcdfd0838a4695cc2e0e705bc0258
Oct 30 20:41:18.223270 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:18.223245    4471 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:41:18.364345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:18.364305    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:18.364854 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:18.364837    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:18.365574 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:18.365553    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:41:19.153737 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:19.153692    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7269-6b3e9a" event={"ID":"119e7954-1273-4cf3-853a-11925bd2a4a0","Type":"ContainerStarted","Data":"ea98116e010d1748481f02ca0bbc0b9b55edcdfd0838a4695cc2e0e705bc0258"}
Oct 30 20:41:22.450576 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:22.439358    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht" podUID="5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:22.501267 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:22.498489    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl" podUID="a4344406-c869-420f-8cf3-93c5e0549be8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:25.388248 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:25.388209    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7269-6b3e9a" event={"ID":"119e7954-1273-4cf3-853a-11925bd2a4a0","Type":"ContainerStarted","Data":"bb4ae9ac0ea789be76e9f47089a7c0b0004912822c849fb224b0d25259ca68fd"}
Oct 30 20:41:25.504615 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:25.504548    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7269-6b3e9a" podStartSLOduration=1.704104129 podStartE2EDuration="8.504528506s" totalImagesPullingTime="6.800424377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:25.50315928 +0000 UTC m=+1305.181787894" watchObservedRunningTime="2026-10-30 20:41:25.504528506 +0000 UTC m=+1305.183157122"
Oct 30 20:41:26.995625 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:26.995582    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7269-6b3e9a"]
Oct 30 20:41:26.999864 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:26.999831    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7269-6b3e9a"]
Oct 30 20:41:27.399449 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:27.399401    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7269-6b3e9a" podUID="119e7954-1273-4cf3-853a-11925bd2a4a0" containerName="test-curl-7269-6b3e9a" containerID="cri-o://bb4ae9ac0ea789be76e9f47089a7c0b0004912822c849fb224b0d25259ca68fd" gracePeriod=2
Oct 30 20:41:27.450058 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:41:27.450015    4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-6b3e9a\" is forbidden: User \"system:node:ip-10-0-138-34.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-34.us-west-1.compute.internal' and this object" podUID="119e7954-1273-4cf3-853a-11925bd2a4a0" pod="opendatahub/test-curl-7269-6b3e9a"
Oct 30 20:41:29.658532 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:29.658461    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-6b3e9a"
Oct 30 20:41:29.771911 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:29.771880    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119e7954-1273-4cf3-853a-11925bd2a4a0-kube-api-access-rq9x5\" (UniqueName: \"kubernetes.io/projected/119e7954-1273-4cf3-853a-11925bd2a4a0-kube-api-access-rq9x5\") pod \"119e7954-1273-4cf3-853a-11925bd2a4a0\" (UID: \"119e7954-1273-4cf3-853a-11925bd2a4a0\") "
Oct 30 20:41:29.774257 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:29.774220    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119e7954-1273-4cf3-853a-11925bd2a4a0-kube-api-access-rq9x5" pod "119e7954-1273-4cf3-853a-11925bd2a4a0" (UID: "119e7954-1273-4cf3-853a-11925bd2a4a0"). InnerVolumeSpecName "kube-api-access-rq9x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:29.873415 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:29.873371    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq9x5\" (UniqueName: \"kubernetes.io/projected/119e7954-1273-4cf3-853a-11925bd2a4a0-kube-api-access-rq9x5\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:41:30.428147 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:30.427864    4471 generic.go:356] "Generic (PLEG): container finished" podID="119e7954-1273-4cf3-853a-11925bd2a4a0" containerID="bb4ae9ac0ea789be76e9f47089a7c0b0004912822c849fb224b0d25259ca68fd" exitCode=137
Oct 30 20:41:30.436290 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:30.436236    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-6b3e9a"
Oct 30 20:41:30.438505 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:30.438464    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="119e7954-1273-4cf3-853a-11925bd2a4a0" path="/var/lib/kubelet/pods/119e7954-1273-4cf3-853a-11925bd2a4a0/volumes"
Oct 30 20:41:30.439521 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:30.439481    4471 scope.go:122] "RemoveContainer" containerID="bb4ae9ac0ea789be76e9f47089a7c0b0004912822c849fb224b0d25259ca68fd"
Oct 30 20:41:30.470771 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:30.470740    4471 scope.go:122] "RemoveContainer" containerID="bb4ae9ac0ea789be76e9f47089a7c0b0004912822c849fb224b0d25259ca68fd"
Oct 30 20:41:30.471355 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:41:30.471264    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb4ae9ac0ea789be76e9f47089a7c0b0004912822c849fb224b0d25259ca68fd\": container with ID starting with bb4ae9ac0ea789be76e9f47089a7c0b0004912822c849fb224b0d25259ca68fd not found: ID does not exist" containerID="bb4ae9ac0ea789be76e9f47089a7c0b0004912822c849fb224b0d25259ca68fd"
Oct 30 20:41:30.471581 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:30.471406    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb4ae9ac0ea789be76e9f47089a7c0b0004912822c849fb224b0d25259ca68fd"} err="failed to get container status \"bb4ae9ac0ea789be76e9f47089a7c0b0004912822c849fb224b0d25259ca68fd\": rpc error: code = NotFound desc = could not find container \"bb4ae9ac0ea789be76e9f47089a7c0b0004912822c849fb224b0d25259ca68fd\": container with ID starting with bb4ae9ac0ea789be76e9f47089a7c0b0004912822c849fb224b0d25259ca68fd not found: ID does not exist"
Oct 30 20:41:30.955242 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:41:30.955207    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:41:37.472900 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.472866    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"]
Oct 30 20:41:37.473300 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.473288    4471 cpu_manager.go:414] "containerMap: removing container" podUID="119e7954-1273-4cf3-853a-11925bd2a4a0" containerName="test-curl-7269-6b3e9a"
Oct 30 20:41:37.473370 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.473301    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="119e7954-1273-4cf3-853a-11925bd2a4a0" containerName="test-curl-7269-6b3e9a"
Oct 30 20:41:37.473416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.473402    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="119e7954-1273-4cf3-853a-11925bd2a4a0" containerName="test-curl-7269-6b3e9a"
Oct 30 20:41:37.477246 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.477227    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.490299 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.490275    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-data\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.490410 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.490347    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7nhz\" (UniqueName: \"kubernetes.io/projected/7b0db74e-64b2-4bb2-83da-ee1669156580-kube-api-access-q7nhz\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.490473 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.490416    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-credential-socket\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.490523 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.490499    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-workload-socket\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.490567 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.490530    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-envoy\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.490567 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.490562    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-workload-certs\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.490665 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.490651    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-podinfo\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.490724 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.490672    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b0db74e-64b2-4bb2-83da-ee1669156580-istiod-ca-cert\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.490778 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.490740    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-token\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.529272 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.529243    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-df031b-openshift-default-dockercfg-8m6p4\""
Oct 30 20:41:37.581275 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.581239    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"]
Oct 30 20:41:37.591835 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.591806    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7nhz\" (UniqueName: \"kubernetes.io/projected/7b0db74e-64b2-4bb2-83da-ee1669156580-kube-api-access-q7nhz\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.592015 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.591856    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-credential-socket\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.592015 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.591897    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-workload-socket\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.592015 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.591932    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-envoy\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.592015 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.591980    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-workload-certs\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.592227 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.592085    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-podinfo\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.592227 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.592117    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b0db74e-64b2-4bb2-83da-ee1669156580-istiod-ca-cert\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.592227 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.592214    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-token\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.592480 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.592454    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-credential-socket\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.592726 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.592704    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-data\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.592965 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.592837    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-workload-certs\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.592965 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.592785    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-workload-socket\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.593135 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.593008    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-data\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.593470 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.593409    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b0db74e-64b2-4bb2-83da-ee1669156580-istiod-ca-cert\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.594975 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.594956    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-podinfo\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.595087 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.594996    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-envoy\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.648693 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.648665    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7nhz\" (UniqueName: \"kubernetes.io/projected/7b0db74e-64b2-4bb2-83da-ee1669156580-kube-api-access-q7nhz\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.648865 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.648695    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-token\") pod \"e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") " pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:37.789651 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:37.789559    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:38.221882 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:38.221848    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"]
Oct 30 20:41:38.223834 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:41:38.223795    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0db74e_64b2_4bb2_83da_ee1669156580.slice/crio-afc317edb761265e22881a0a4ddad25bcf0a761d495e73b2283c5fcfee31dc16 WatchSource:0}: Error finding container afc317edb761265e22881a0a4ddad25bcf0a761d495e73b2283c5fcfee31dc16: Status 404 returned error can't find the container with id afc317edb761265e22881a0a4ddad25bcf0a761d495e73b2283c5fcfee31dc16
Oct 30 20:41:38.226494 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:38.226455    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:41:38.226595 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:38.226547    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:41:38.226595 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:38.226577    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:41:38.473502 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:38.473410    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp" event={"ID":"7b0db74e-64b2-4bb2-83da-ee1669156580","Type":"ContainerStarted","Data":"b392525196987dcd71139ce2bfca3ce121ce3c205f3aedc992387fa888915756"}
Oct 30 20:41:38.473502 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:38.473462    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp" event={"ID":"7b0db74e-64b2-4bb2-83da-ee1669156580","Type":"ContainerStarted","Data":"afc317edb761265e22881a0a4ddad25bcf0a761d495e73b2283c5fcfee31dc16"}
Oct 30 20:41:38.595886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:38.595825    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp" podStartSLOduration=1.5958047020000001 podStartE2EDuration="1.595804702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:38.591270333 +0000 UTC m=+1318.269898949" watchObservedRunningTime="2026-10-30 20:41:38.595804702 +0000 UTC m=+1318.274433319"
Oct 30 20:41:38.789741 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:38.789647    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:39.792629 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:39.792592    4471 patch_prober.go:29] interesting pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 20:41:39.793047 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:39.792661    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp" podUID="7b0db74e-64b2-4bb2-83da-ee1669156580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 20:41:40.378353 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.360160    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"]
Oct 30 20:41:40.399347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.379863    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.460749 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.401884    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"]
Oct 30 20:41:40.545345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.532637    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-data\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.545345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.532705    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-workload-socket\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.545345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.532755    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clplh\" (UniqueName: \"kubernetes.io/projected/d03eaf5f-735f-43cf-aa21-45901e9e6c04-kube-api-access-clplh\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.545345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.532801    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-token\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.545345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.532862    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-credential-socket\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.545345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.532885    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istiod-ca-cert\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.545345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.532931    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-envoy\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.545345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.532964    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-workload-certs\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.545345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.533005    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-podinfo\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.545345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.533270    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8e2fc1-openshift-default-dockercfg-77c66\""
Oct 30 20:41:40.569467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.569440    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"]
Oct 30 20:41:40.575403 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.575380    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"]
Oct 30 20:41:40.575685 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.575672    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.634780 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.634695    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clplh\" (UniqueName: \"kubernetes.io/projected/d03eaf5f-735f-43cf-aa21-45901e9e6c04-kube-api-access-clplh\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.634937 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.634821    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-token\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.634937 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.634907    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-credential-socket\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.635092 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.634932    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istiod-ca-cert\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.635092 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.634998    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-envoy\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.635092 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.635031    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-workload-certs\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.635092 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.635085    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-podinfo\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.635311 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.635224    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-data\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.635311 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.635262    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-credential-socket\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.635311 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.635308    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-workload-socket\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.635578 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.635555    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-data\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.635641 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.635605    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-workload-socket\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.636010 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.635989    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-workload-certs\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.636255 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.636228    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istiod-ca-cert\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.638573 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.638551    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-envoy\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.641795 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.641767    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-podinfo\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.776650 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.776112    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clplh\" (UniqueName: \"kubernetes.io/projected/d03eaf5f-735f-43cf-aa21-45901e9e6c04-kube-api-access-clplh\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.793349 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.784921    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ftmw\" (UniqueName: \"kubernetes.io/projected/a7c2de2a-ca80-4191-af39-44f6d686a52e-kube-api-access-4ftmw\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.793349 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.785004    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-podinfo\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.793349 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.785092    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-credential-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.793349 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.785209    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-token\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.793349 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.785243    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-workload-certs\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.793349 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.785449    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c2de2a-ca80-4191-af39-44f6d686a52e-istiod-ca-cert\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.793349 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.785503    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-data\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.793349 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.785535    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-workload-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.793349 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.785600    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-envoy\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.793349 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.787544    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-token\") pod \"e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") " pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.796277 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.795774    4471 patch_prober.go:29] interesting pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 20:41:40.796277 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.795832    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp" podUID="7b0db74e-64b2-4bb2-83da-ee1669156580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 20:41:40.876382 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.873566    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:40.887197 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.887121    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-workload-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.887197 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.887169    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-envoy\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.887408 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.887257    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ftmw\" (UniqueName: \"kubernetes.io/projected/a7c2de2a-ca80-4191-af39-44f6d686a52e-kube-api-access-4ftmw\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.887408 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.887308    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-podinfo\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.887408 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.887401    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-credential-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.887823 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.887788    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-workload-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.887926 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.887838    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-credential-socket\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.888124 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.887997    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-token\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.888699 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.888673    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-workload-certs\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.889019 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.888999    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-workload-certs\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.889163 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.889149    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c2de2a-ca80-4191-af39-44f6d686a52e-istiod-ca-cert\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.889236 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.889191    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-data\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.889529 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.889508    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-data\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.890093 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.890061    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c2de2a-ca80-4191-af39-44f6d686a52e-istiod-ca-cert\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.890701 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.890584    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-podinfo\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.890885 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.890859    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-envoy\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.988371 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.987854    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-token\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:40.995086 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:40.995006    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ftmw\" (UniqueName: \"kubernetes.io/projected/a7c2de2a-ca80-4191-af39-44f6d686a52e-kube-api-access-4ftmw\") pod \"e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") " pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:41.212397 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:41.208703    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:41.443581 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:41.439951    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"]
Oct 30 20:41:41.534626 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:41.509105    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:41:41.534626 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:41.509270    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:41:41.593193 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:41.587180    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:41:41.601915 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:41.601884    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7" event={"ID":"d03eaf5f-735f-43cf-aa21-45901e9e6c04","Type":"ContainerStarted","Data":"339beda8334f65873c99df4f37a594204baeaeb801095556764a78052c6c4d98"}
Oct 30 20:41:41.790139 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:41.790043    4471 patch_prober.go:29] interesting pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 20:41:41.790139 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:41.790095    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp" podUID="7b0db74e-64b2-4bb2-83da-ee1669156580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 20:41:41.829288 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:41.829265    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"]
Oct 30 20:41:41.831738 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:41:41.831714    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c2de2a_ca80_4191_af39_44f6d686a52e.slice/crio-06f6dc2dccb85ebeab00cd43386cd4f614da987828517a319299b2c2e4eb2588 WatchSource:0}: Error finding container 06f6dc2dccb85ebeab00cd43386cd4f614da987828517a319299b2c2e4eb2588: Status 404 returned error can't find the container with id 06f6dc2dccb85ebeab00cd43386cd4f614da987828517a319299b2c2e4eb2588
Oct 30 20:41:41.835178 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:41.834991    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:41:41.835178 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:41.835082    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:41:41.835178 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:41.835132    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:41:42.615347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:42.615118    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7" event={"ID":"d03eaf5f-735f-43cf-aa21-45901e9e6c04","Type":"ContainerStarted","Data":"a9ba5e64b7f49212415bc458bd5f924389f7ad6b78f6b828640773f77becd359"}
Oct 30 20:41:42.617352 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:42.617280    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb" event={"ID":"a7c2de2a-ca80-4191-af39-44f6d686a52e","Type":"ContainerStarted","Data":"fb14458278243aae1562baca64dd110d3f59f0e492ffd5e9ae6a1378a341ff56"}
Oct 30 20:41:42.617490 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:42.617354    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb" event={"ID":"a7c2de2a-ca80-4191-af39-44f6d686a52e","Type":"ContainerStarted","Data":"06f6dc2dccb85ebeab00cd43386cd4f614da987828517a319299b2c2e4eb2588"}
Oct 30 20:41:42.753347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:42.748557    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7" podStartSLOduration=2.748538142 podStartE2EDuration="2.748538142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:42.747852904 +0000 UTC m=+1322.426481518" watchObservedRunningTime="2026-10-30 20:41:42.748538142 +0000 UTC m=+1322.427166759"
Oct 30 20:41:42.791349 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:42.790721    4471 patch_prober.go:29] interesting pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 20:41:42.791349 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:42.790783    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp" podUID="7b0db74e-64b2-4bb2-83da-ee1669156580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 20:41:42.894350 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:42.882698    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:42.894350 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:42.882821    4471 patch_prober.go:29] interesting pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 20:41:42.894350 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:42.882863    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7" podUID="d03eaf5f-735f-43cf-aa21-45901e9e6c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 20:41:43.213410 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:43.212400    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:43.213410 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:43.212655    4471 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 20:41:43.213410 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:43.212724    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb" podUID="a7c2de2a-ca80-4191-af39-44f6d686a52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 20:41:43.802354 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:43.802124    4471 patch_prober.go:29] interesting pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 20:41:43.802354 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:43.802183    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp" podUID="7b0db74e-64b2-4bb2-83da-ee1669156580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 20:41:43.874752 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:43.874709    4471 patch_prober.go:29] interesting pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 20:41:43.875027 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:43.874999    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7" podUID="d03eaf5f-735f-43cf-aa21-45901e9e6c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 20:41:44.210344 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:44.209774    4471 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 20:41:44.210344 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:44.209838    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb" podUID="a7c2de2a-ca80-4191-af39-44f6d686a52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 20:41:44.790229 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:44.790192    4471 patch_prober.go:29] interesting pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 20:41:44.790589 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:44.790534    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp" podUID="7b0db74e-64b2-4bb2-83da-ee1669156580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 20:41:44.875789 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:44.875750    4471 patch_prober.go:29] interesting pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 20:41:44.876089 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:44.876059    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7" podUID="d03eaf5f-735f-43cf-aa21-45901e9e6c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 20:41:45.210345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:45.209754    4471 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 20:41:45.210345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:45.209834    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb" podUID="a7c2de2a-ca80-4191-af39-44f6d686a52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 20:41:45.875179 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:45.874697    4471 patch_prober.go:29] interesting pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 20:41:45.875179 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:45.874766    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7" podUID="d03eaf5f-735f-43cf-aa21-45901e9e6c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 20:41:46.212582 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:46.211677    4471 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 20:41:46.212582 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:46.211741    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb" podUID="a7c2de2a-ca80-4191-af39-44f6d686a52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 20:41:46.856351 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:46.853426    4471 patch_prober.go:29] interesting pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:41:46.856351 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:46.853493    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp" podUID="7b0db74e-64b2-4bb2-83da-ee1669156580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:46.876821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:46.876779    4471 patch_prober.go:29] interesting pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 20:41:46.877104 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:46.877064    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7" podUID="d03eaf5f-735f-43cf-aa21-45901e9e6c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 20:41:47.003491 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:47.003454    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:47.005285 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:47.005265    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:47.020760 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:47.020735    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:41:47.160358 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:47.155021    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb" podStartSLOduration=7.155002509 podStartE2EDuration="7.155002509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:42.880674732 +0000 UTC m=+1322.559303349" watchObservedRunningTime="2026-10-30 20:41:47.155002509 +0000 UTC m=+1326.833631127"
Oct 30 20:41:47.248369 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:47.230824    4471 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 20:41:47.248369 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:47.230927    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb" podUID="a7c2de2a-ca80-4191-af39-44f6d686a52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 20:41:47.879869 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:47.879364    4471 patch_prober.go:29] interesting pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 20:41:47.879869 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:47.879439    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7" podUID="d03eaf5f-735f-43cf-aa21-45901e9e6c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 20:41:48.209832 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:48.209743    4471 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 20:41:48.209832 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:48.209805    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb" podUID="a7c2de2a-ca80-4191-af39-44f6d686a52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 20:41:48.875120 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:48.875079    4471 patch_prober.go:29] interesting pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 30 20:41:48.875547 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:48.875146    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7" podUID="d03eaf5f-735f-43cf-aa21-45901e9e6c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 30 20:41:49.209707 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:49.209611    4471 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 20:41:49.209707 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:49.209687    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb" podUID="a7c2de2a-ca80-4191-af39-44f6d686a52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 20:41:50.012465 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:50.012429    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:50.013754 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:50.013729    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:50.014434 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:50.014417    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:41:50.213703 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:50.213674    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:50.214003 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:50.213980    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:50.214779 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:50.214761    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:41:52.524457 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.524405    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"]
Oct 30 20:41:52.534717 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.534684    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.555775 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.555747    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"]
Oct 30 20:41:52.559412 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.559394    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.577486 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.577452    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165b506b-c809-43eb-a183-686869cf35db-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.577614 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.577491    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.577614 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.577516    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.577614 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.577606    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165b506b-c809-43eb-a183-686869cf35db-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.577771 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.577654    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmfl7\" (UniqueName: \"kubernetes.io/projected/165b506b-c809-43eb-a183-686869cf35db-kube-api-access-pmfl7\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.577771 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.577676    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.577771 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.577700    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.577866 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.577805    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.577866 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.577832    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.577866 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.577855    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.577986 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.577917    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.578055 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.578039    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6dlm\" (UniqueName: \"kubernetes.io/projected/b2d2f36f-2a1b-402b-8a71-08f174e078c4-kube-api-access-x6dlm\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.578144 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.578130    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.578202 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.578164    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.578202 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.578186    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.578294 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.578211    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.578294 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.578288    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165b506b-c809-43eb-a183-686869cf35db-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.578429 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.578309    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.787428 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.787295    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.787881 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.787742    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.788064 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.788039    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.788193 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.788174    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.788354 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.788339    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165b506b-c809-43eb-a183-686869cf35db-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.788463 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.788446    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.788628 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.788611    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165b506b-c809-43eb-a183-686869cf35db-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.788768 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.788752    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.788876 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.788862    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.788976 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.788962    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165b506b-c809-43eb-a183-686869cf35db-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.789085 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.789072    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmfl7\" (UniqueName: \"kubernetes.io/projected/165b506b-c809-43eb-a183-686869cf35db-kube-api-access-pmfl7\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.789176 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.789163    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.789292 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.789279    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.789426 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.789410    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.789537 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.789521    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.789708 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.789689    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.789850 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.789830    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.790041 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.790027    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6dlm\" (UniqueName: \"kubernetes.io/projected/b2d2f36f-2a1b-402b-8a71-08f174e078c4-kube-api-access-x6dlm\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.790821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.790788    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-workload-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.797721 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.797680    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165b506b-c809-43eb-a183-686869cf35db-istiod-ca-cert\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.799659 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.799246    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.814348 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.802888    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.814572 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.808911    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.814719 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.809175    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.822343 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.810469    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.837305 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.822884    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-credential-socket\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.837820 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.825915    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-workload-certs\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.837968 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.837744    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-istio-data\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.840892 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.840865    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"]
Oct 30 20:41:52.842815 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.842790    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165b506b-c809-43eb-a183-686869cf35db-istio-podinfo\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.843283 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.843237    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-istio-envoy\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.843860 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.843839    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.844960 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.844922    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.850099 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.850075    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"]
Oct 30 20:41:52.851584 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.851546    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmfl7\" (UniqueName: \"kubernetes.io/projected/165b506b-c809-43eb-a183-686869cf35db-kube-api-access-pmfl7\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:52.865355 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.865315    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.865555 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.865535    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6dlm\" (UniqueName: \"kubernetes.io/projected/b2d2f36f-2a1b-402b-8a71-08f174e078c4-kube-api-access-x6dlm\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.876393 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.874574    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:52.876944 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:52.876925    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165b506b-c809-43eb-a183-686869cf35db-istio-token\") pod \"e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") " pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:53.156272 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:53.156241    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:53.312431 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:53.312408    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"]
Oct 30 20:41:53.314650 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:41:53.314620    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d2f36f_2a1b_402b_8a71_08f174e078c4.slice/crio-c7672ab18a035a6cf9e5af1bef6385cab8665f203c31990f2dc08e0d4df2f195 WatchSource:0}: Error finding container c7672ab18a035a6cf9e5af1bef6385cab8665f203c31990f2dc08e0d4df2f195: Status 404 returned error can't find the container with id c7672ab18a035a6cf9e5af1bef6385cab8665f203c31990f2dc08e0d4df2f195
Oct 30 20:41:53.317247 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:53.317216    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:41:53.317315 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:53.317297    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:41:53.317378 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:53.317349    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:41:53.546944 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:53.546921    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"]
Oct 30 20:41:53.549505 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:41:53.549478    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod165b506b_c809_43eb_a183_686869cf35db.slice/crio-2c734bc46c482a49741f8cb044a38227d743e770b33888664ac9ac65eca60982 WatchSource:0}: Error finding container 2c734bc46c482a49741f8cb044a38227d743e770b33888664ac9ac65eca60982: Status 404 returned error can't find the container with id 2c734bc46c482a49741f8cb044a38227d743e770b33888664ac9ac65eca60982
Oct 30 20:41:53.552271 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:53.552237    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:41:53.552385 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:53.552308    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:41:53.552385 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:53.552365    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:41:53.917378 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:53.916434    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw" event={"ID":"165b506b-c809-43eb-a183-686869cf35db","Type":"ContainerStarted","Data":"402de59011fb87953260c8e332542a0e94fff69cb91b94ecb368dde240f1ae31"}
Oct 30 20:41:53.917378 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:53.916503    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw" event={"ID":"165b506b-c809-43eb-a183-686869cf35db","Type":"ContainerStarted","Data":"2c734bc46c482a49741f8cb044a38227d743e770b33888664ac9ac65eca60982"}
Oct 30 20:41:53.922767 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:53.922189    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng" event={"ID":"b2d2f36f-2a1b-402b-8a71-08f174e078c4","Type":"ContainerStarted","Data":"cf7de9fd00ba72ca142b8b069bda4fd3bb487a0003569255e6545f98977dffe5"}
Oct 30 20:41:53.922767 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:53.922257    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng" event={"ID":"b2d2f36f-2a1b-402b-8a71-08f174e078c4","Type":"ContainerStarted","Data":"c7672ab18a035a6cf9e5af1bef6385cab8665f203c31990f2dc08e0d4df2f195"}
Oct 30 20:41:54.065350 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:54.051647    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw" podStartSLOduration=2.05162176 podStartE2EDuration="2.05162176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:54.041709505 +0000 UTC m=+1333.720338123" watchObservedRunningTime="2026-10-30 20:41:54.05162176 +0000 UTC m=+1333.730250395"
Oct 30 20:41:54.172466 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:54.164686    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:41:54.173558 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:54.173516    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 20:41:54.173718 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:54.173581    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw" podUID="165b506b-c809-43eb-a183-686869cf35db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 20:41:54.878354 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:54.877738    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:41:54.882351 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:54.880350    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 20:41:54.882351 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:54.880410    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng" podUID="b2d2f36f-2a1b-402b-8a71-08f174e078c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 20:41:55.166409 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.158725    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 20:41:55.166409 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.158790    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw" podUID="165b506b-c809-43eb-a183-686869cf35db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 20:41:55.363348 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.362503    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng" podStartSLOduration=3.362480047 podStartE2EDuration="3.362480047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:54.173728378 +0000 UTC m=+1333.852357005" watchObservedRunningTime="2026-10-30 20:41:55.362480047 +0000 UTC m=+1335.041108673"
Oct 30 20:41:55.391349 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.389878    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"]
Oct 30 20:41:55.407345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.407177    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.440064 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.439996    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"]
Oct 30 20:41:55.560487 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.560449    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.564446 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.564394    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ght5p\" (UniqueName: \"kubernetes.io/projected/03e1b048-5b3f-43fd-b321-897f8d6e6e53-kube-api-access-ght5p\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.564802 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.564765    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.565119 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.565002    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.569476 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.569446    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.569804 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.569782    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.600849 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.600800    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.603190 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.603152    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.603671 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.603638    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.742583 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.733092    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.761487 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.761448    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.767525 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.752857    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-workload-certs\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.778387 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.767831    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.778387 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.768252    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-credential-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.779499 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.778933    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.779499 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.779042    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ght5p\" (UniqueName: \"kubernetes.io/projected/03e1b048-5b3f-43fd-b321-897f8d6e6e53-kube-api-access-ght5p\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.779499 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.779160    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.779499 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.779228    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.779499 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.779315    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.779499 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.779459    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.797804 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.795843    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-data\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.797804 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.797384    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istiod-ca-cert\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.805844 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.805660    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-workload-socket\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.870417 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.859633    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-envoy\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.888347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.886754    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-podinfo\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.888347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.888129    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 20:41:55.888347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.888191    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng" podUID="b2d2f36f-2a1b-402b-8a71-08f174e078c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 20:41:55.890617 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.889817    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-token\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:55.909217 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:55.909190    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ght5p\" (UniqueName: \"kubernetes.io/projected/03e1b048-5b3f-43fd-b321-897f8d6e6e53-kube-api-access-ght5p\") pod \"e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") " pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:56.027803 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:56.027692    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:56.164347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:56.162617    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 20:41:56.164347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:56.162723    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw" podUID="165b506b-c809-43eb-a183-686869cf35db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 20:41:56.719360 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:56.673016    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh" podUID="3dcbb5f5-93f4-4210-8274-2278fb9924ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:56.914768 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:56.914732    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 20:41:56.915390 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:56.915360    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng" podUID="b2d2f36f-2a1b-402b-8a71-08f174e078c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 20:41:56.998384 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:56.971884    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7" podUID="02b9bb13-c775-44a5-b034-563359097687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 20:41:57.151388 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:57.151346    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk" podUID="a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:57.163461 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:57.163414    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 20:41:57.163722 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:57.163671    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw" podUID="165b506b-c809-43eb-a183-686869cf35db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 20:41:57.313022 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:57.312992    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"]
Oct 30 20:41:57.366575 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:57.366516    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:41:57.367049 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:57.367020    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:41:57.367438 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:57.367262    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:41:57.439781 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:57.439739    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht" podUID="5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:57.460757 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:57.459195    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl" podUID="a4344406-c869-420f-8cf3-93c5e0549be8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:57.884197 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:57.877274    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4" podUID="0ec28e09-b926-4e73-8f40-2f1ab3a41df8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 20:41:57.888036 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:57.884780    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf" podUID="bf89b9b8-0419-48bc-b410-12807dab75db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:57.900786 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:57.896407    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh" podUID="3dcbb5f5-93f4-4210-8274-2278fb9924ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:57.900786 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:57.896796    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:41:57.900786 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:57.898303    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 20:41:57.900786 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:57.898360    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng" podUID="b2d2f36f-2a1b-402b-8a71-08f174e078c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 20:41:58.040362 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:58.040208    4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk" podUID="a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:58.049347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:58.047168    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7" podUID="02b9bb13-c775-44a5-b034-563359097687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:58.049347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:58.047600    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:41:58.224399 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:58.178275    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 20:41:58.224399 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:58.178361    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw" podUID="165b506b-c809-43eb-a183-686869cf35db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 20:41:58.224399 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:58.178440    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk" podUID="a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:58.224399 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:58.181665    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:41:58.263388 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:58.251967    4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf" podUID="bf89b9b8-0419-48bc-b410-12807dab75db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:58.505410 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:58.464861    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl" podUID="a4344406-c869-420f-8cf3-93c5e0549be8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 20:41:58.505410 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:58.466084    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:41:58.505410 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:58.488595    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht" podUID="5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 20:41:58.544367 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:58.538616    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524" podUID="31cc010d-9e1f-42c2-a736-0ceacbedb8c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:58.581730 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:58.581690    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:41:58.946432 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:58.807170    4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524" podUID="31cc010d-9e1f-42c2-a736-0ceacbedb8c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": context deadline exceeded"
Oct 30 20:41:58.946432 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:58.922517    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b" podUID="1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 20:41:58.946432 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:58.922479    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf" podUID="bf89b9b8-0419-48bc-b410-12807dab75db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 20:41:59.027783 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.027715    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh" podUID="3dcbb5f5-93f4-4210-8274-2278fb9924ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:59.029252 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.029220    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4" podUID="0ec28e09-b926-4e73-8f40-2f1ab3a41df8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:59.127347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.106147    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 20:41:59.128121 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.128085    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng" podUID="b2d2f36f-2a1b-402b-8a71-08f174e078c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 20:41:59.138347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.128835    4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh" podUID="3dcbb5f5-93f4-4210-8274-2278fb9924ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:59.250306 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.232503    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk" podUID="a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:59.250306 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.232897    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7" podUID="02b9bb13-c775-44a5-b034-563359097687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 20:41:59.250306 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.244752    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr" event={"ID":"03e1b048-5b3f-43fd-b321-897f8d6e6e53","Type":"ContainerStarted","Data":"cb0cfeec4ea49952fc4fa219e6e1137d91271b77ce04146f57e33e184851e66f"}
Oct 30 20:41:59.250306 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.244834    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr" event={"ID":"03e1b048-5b3f-43fd-b321-897f8d6e6e53","Type":"ContainerStarted","Data":"7984b9c417c198ddde9f583323adfda8ede60e5328004bfd7d4572bacf6de649"}
Oct 30 20:41:59.250306 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.244931    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:41:59.250306 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.244999    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:41:59.250306 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.245048    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:41:59.340188 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.340151    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 20:41:59.361403 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.354971    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw" podUID="165b506b-c809-43eb-a183-686869cf35db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 20:41:59.411388 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.407678    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46" podUID="13896eb7-f9cc-4cf2-9364-b723651037af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:59.478347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.474441    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl" podUID="a4344406-c869-420f-8cf3-93c5e0549be8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:59.531408 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.520531    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 20:41:59.531408 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.520583    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr" podUID="03e1b048-5b3f-43fd-b321-897f8d6e6e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 20:41:59.531408 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.521070    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht" podUID="5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 20:41:59.560465 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.560413    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524" podUID="31cc010d-9e1f-42c2-a736-0ceacbedb8c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:41:59.561134 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.561110    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:41:59.896507 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.896473    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 20:41:59.896791 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.896747    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng" podUID="b2d2f36f-2a1b-402b-8a71-08f174e078c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 20:42:00.086445 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:41:59.954405    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b" podUID="1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:00.094670 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:00.040425    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4" podUID="0ec28e09-b926-4e73-8f40-2f1ab3a41df8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:00.107737 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:00.041213    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 20:42:00.107978 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:00.107948    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr" podUID="03e1b048-5b3f-43fd-b321-897f8d6e6e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 20:42:00.108103 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:00.041276    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh" podUID="3dcbb5f5-93f4-4210-8274-2278fb9924ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:00.113360 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:00.041251    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf" podUID="bf89b9b8-0419-48bc-b410-12807dab75db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:00.145408 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:00.095135    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:42:00.160018 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:00.159825    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 20:42:00.160018 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:00.159882    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw" podUID="165b506b-c809-43eb-a183-686869cf35db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 20:42:00.202987 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:00.202927    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr" podStartSLOduration=5.202906989 podStartE2EDuration="5.202906989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:59.683444051 +0000 UTC m=+1339.362072704" watchObservedRunningTime="2026-10-30 20:42:00.202906989 +0000 UTC m=+1339.881535607"
Oct 30 20:42:00.282345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:00.282134    4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46" podUID="13896eb7-f9cc-4cf2-9364-b723651037af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:00.282345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:00.282239    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk" podUID="a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 20:42:00.282618 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:00.282408    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7" podUID="02b9bb13-c775-44a5-b034-563359097687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 20:42:00.361353 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:00.349783    4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7" podUID="02b9bb13-c775-44a5-b034-563359097687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:00.424987 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:00.421853    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46" podUID="13896eb7-f9cc-4cf2-9364-b723651037af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:00.424987 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:00.422223    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:42:00.459481 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:00.459413    4471 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","besteffort","pod119e7954-1273-4cf3-853a-11925bd2a4a0"] err="unable to destroy cgroup paths for cgroup [kubepods besteffort pod119e7954-1273-4cf3-853a-11925bd2a4a0] : Timed out while waiting for systemd to remove kubepods-besteffort-pod119e7954_1273_4cf3_853a_11925bd2a4a0.slice"
Oct 30 20:42:00.459750 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:42:00.459720    4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods besteffort pod119e7954-1273-4cf3-853a-11925bd2a4a0] : unable to destroy cgroup paths for cgroup [kubepods besteffort pod119e7954-1273-4cf3-853a-11925bd2a4a0] : Timed out while waiting for systemd to remove kubepods-besteffort-pod119e7954_1273_4cf3_853a_11925bd2a4a0.slice" pod="opendatahub/test-curl-7269-6b3e9a" podUID="119e7954-1273-4cf3-853a-11925bd2a4a0"
Oct 30 20:42:00.557378 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:00.526120    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht" podUID="5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:00.557378 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:00.527375    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl" podUID="a4344406-c869-420f-8cf3-93c5e0549be8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:00.608455 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:00.589601    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524" podUID="31cc010d-9e1f-42c2-a736-0ceacbedb8c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 20:42:00.882453 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:00.882402    4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl" podUID="a4344406-c869-420f-8cf3-93c5e0549be8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Oct 30 20:42:01.073420 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:01.073382    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 20:42:01.073738 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:01.073708    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng" podUID="b2d2f36f-2a1b-402b-8a71-08f174e078c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 20:42:01.088520 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:01.088466    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b" podUID="1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 20:42:01.116794 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:01.107496    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4" podUID="0ec28e09-b926-4e73-8f40-2f1ab3a41df8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:01.116794 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:01.108255    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh" podUID="3dcbb5f5-93f4-4210-8274-2278fb9924ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:01.117040 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:01.114652    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf" podUID="bf89b9b8-0419-48bc-b410-12807dab75db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:01.124650 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:01.120075    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-6b3e9a"
Oct 30 20:42:01.154835 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:01.147782    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 20:42:01.154835 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:01.147836    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr" podUID="03e1b048-5b3f-43fd-b321-897f8d6e6e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 20:42:01.159930 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:01.159898    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 20:42:01.175801 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:01.175765    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw" podUID="165b506b-c809-43eb-a183-686869cf35db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 20:42:01.297475 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:01.297417    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk" podUID="a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:01.298478 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:01.297758    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7" podUID="02b9bb13-c775-44a5-b034-563359097687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:01.522452 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:01.493525    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46" podUID="13896eb7-f9cc-4cf2-9364-b723651037af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Oct 30 20:42:01.571035 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:42:01.567687    4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-6b3e9a\" is forbidden: User \"system:node:ip-10-0-138-34.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-34.us-west-1.compute.internal' and this object" podUID="119e7954-1273-4cf3-853a-11925bd2a4a0" pod="opendatahub/test-curl-7269-6b3e9a"
Oct 30 20:42:01.571035 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:01.570012    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl" podUID="a4344406-c869-420f-8cf3-93c5e0549be8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:01.577702 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:01.571765    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht" podUID="5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 20:42:01.623472 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:01.616463    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524" podUID="31cc010d-9e1f-42c2-a736-0ceacbedb8c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:01.638360 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:42:01.633166    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:42:01.920354 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:01.919045    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 20:42:01.920354 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:01.919126    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng" podUID="b2d2f36f-2a1b-402b-8a71-08f174e078c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 20:42:02.032158 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:02.031443    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 20:42:02.032499 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:02.032410    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr" podUID="03e1b048-5b3f-43fd-b321-897f8d6e6e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 20:42:02.120471 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:02.120421    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf" podUID="bf89b9b8-0419-48bc-b410-12807dab75db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:02.123053 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:02.123014    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b" podUID="1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 20:42:02.124335 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:02.124288    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh" podUID="3dcbb5f5-93f4-4210-8274-2278fb9924ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 20:42:02.125486 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:02.125464    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4" podUID="0ec28e09-b926-4e73-8f40-2f1ab3a41df8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 20:42:02.160784 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:02.160736    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 20:42:02.161419 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:02.161388    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw" podUID="165b506b-c809-43eb-a183-686869cf35db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 20:42:02.300563 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:02.300447    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk" podUID="a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 20:42:02.303343 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:02.303278    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7" podUID="02b9bb13-c775-44a5-b034-563359097687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:02.496269 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:02.496221    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46" podUID="13896eb7-f9cc-4cf2-9364-b723651037af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Oct 30 20:42:02.572499 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:02.572400    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht" podUID="5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:02.573047 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:02.573015    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl" podUID="a4344406-c869-420f-8cf3-93c5e0549be8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 20:42:02.618545 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:02.618483    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524" podUID="31cc010d-9e1f-42c2-a736-0ceacbedb8c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:02.685584 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:02.685477    4471 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht" podUID="5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:02.730180 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:02.730150    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 20:42:02.882344 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:02.879690    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 20:42:02.882344 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:02.879752    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng" podUID="b2d2f36f-2a1b-402b-8a71-08f174e078c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 20:42:03.037349 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:03.032195    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 20:42:03.037349 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:03.032251    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr" podUID="03e1b048-5b3f-43fd-b321-897f8d6e6e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 20:42:03.140816 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:03.133102    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 20:42:03.140816 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:03.140543    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4" podUID="0ec28e09-b926-4e73-8f40-2f1ab3a41df8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:03.146644 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:03.145305    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b" podUID="1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 30 20:42:03.146644 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:03.145619    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh" podUID="3dcbb5f5-93f4-4210-8274-2278fb9924ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Oct 30 20:42:03.159306 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:03.157586    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 20:42:03.159306 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:03.157641    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw" podUID="165b506b-c809-43eb-a183-686869cf35db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 20:42:03.330864 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:03.316157    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7" podUID="02b9bb13-c775-44a5-b034-563359097687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 20:42:03.330864 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:03.318076    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk" podUID="a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 20:42:03.535341 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:03.535190    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:42:03.575071 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:03.575027    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl" podUID="a4344406-c869-420f-8cf3-93c5e0549be8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:03.575688 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:03.575655    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht" podUID="5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:03.619367 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:03.619241    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524" podUID="31cc010d-9e1f-42c2-a736-0ceacbedb8c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:03.835130 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:03.832873    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 20:42:03.850353 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:03.840669    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 20:42:03.973348 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:03.939030    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 20:42:03.973660 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:03.973618    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng" podUID="b2d2f36f-2a1b-402b-8a71-08f174e078c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 20:42:04.052844 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:04.007246    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 20:42:04.100960 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:04.099439    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 20:42:04.100960 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:04.099497    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr" podUID="03e1b048-5b3f-43fd-b321-897f8d6e6e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 20:42:04.127427 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:04.115650    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:42:04.144586 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:04.127839    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 20:42:04.144586 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:04.142400    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf" podUID="bf89b9b8-0419-48bc-b410-12807dab75db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:04.203706 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:04.203673    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 20:42:04.203957 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:04.203932    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw" podUID="165b506b-c809-43eb-a183-686869cf35db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 20:42:04.204570 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:04.204547    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 20:42:04.230412 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:04.212253    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 20:42:05.028691 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:05.028653    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 20:42:05.029310 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:05.029277    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr" podUID="03e1b048-5b3f-43fd-b321-897f8d6e6e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 20:42:05.165347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:05.164740    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 20:42:05.165347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:05.164805    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw" podUID="165b506b-c809-43eb-a183-686869cf35db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 20:42:05.944538 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:05.944499    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:42:05.944853 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:05.944805    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng" podUID="b2d2f36f-2a1b-402b-8a71-08f174e078c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:06.028363 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:06.028306    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 20:42:06.028524 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:06.028382    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr" podUID="03e1b048-5b3f-43fd-b321-897f8d6e6e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 20:42:06.647129 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:06.647092    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:42:06.647740 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:06.647716    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:42:06.655372 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:06.655350    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:42:07.038889 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:07.038753    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 20:42:07.039165 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:07.039133    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr" podUID="03e1b048-5b3f-43fd-b321-897f8d6e6e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 20:42:07.200533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:07.200490    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:42:07.200700 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:07.200559    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw" podUID="165b506b-c809-43eb-a183-686869cf35db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:07.401631 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:07.401595    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:42:07.402654 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:07.402630    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:42:07.427809 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:07.427785    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:42:08.033490 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:08.033460    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:42:08.033935 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:08.033919    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:42:08.034644 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:08.034622    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:42:13.546032 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:13.545999    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"]
Oct 30 20:42:13.546501 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:13.546183    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg" podUID="69467a1c-7315-4154-85a4-72473a1ecd9d" containerName="istio-proxy" containerID="cri-o://87c6e80a103f5b3b971b5012e650c4d518458f0f13874bb342bfe470317df0bb" gracePeriod=30
Oct 30 20:42:18.803505 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.803477    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:42:18.895459 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.895435    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-data\") pod \"69467a1c-7315-4154-85a4-72473a1ecd9d\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") "
Oct 30 20:42:18.895617 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.895489    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-workload-socket\") pod \"69467a1c-7315-4154-85a4-72473a1ecd9d\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") "
Oct 30 20:42:18.895617 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.895523    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-workload-certs\") pod \"69467a1c-7315-4154-85a4-72473a1ecd9d\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") "
Oct 30 20:42:18.895617 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.895586    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69467a1c-7315-4154-85a4-72473a1ecd9d-kube-api-access-9cb4f\" (UniqueName: \"kubernetes.io/projected/69467a1c-7315-4154-85a4-72473a1ecd9d-kube-api-access-9cb4f\") pod \"69467a1c-7315-4154-85a4-72473a1ecd9d\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") "
Oct 30 20:42:18.895749 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.895625    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-credential-socket\") pod \"69467a1c-7315-4154-85a4-72473a1ecd9d\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") "
Oct 30 20:42:18.895749 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.895657    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69467a1c-7315-4154-85a4-72473a1ecd9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69467a1c-7315-4154-85a4-72473a1ecd9d-istiod-ca-cert\") pod \"69467a1c-7315-4154-85a4-72473a1ecd9d\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") "
Oct 30 20:42:18.895749 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.895692    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-envoy\") pod \"69467a1c-7315-4154-85a4-72473a1ecd9d\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") "
Oct 30 20:42:18.895879 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.895761    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-podinfo\") pod \"69467a1c-7315-4154-85a4-72473a1ecd9d\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") "
Oct 30 20:42:18.895879 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.895770    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-data" pod "69467a1c-7315-4154-85a4-72473a1ecd9d" (UID: "69467a1c-7315-4154-85a4-72473a1ecd9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:18.895879 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.895801    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-workload-socket" pod "69467a1c-7315-4154-85a4-72473a1ecd9d" (UID: "69467a1c-7315-4154-85a4-72473a1ecd9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:18.895879 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.895824    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-token\" (UniqueName: \"kubernetes.io/projected/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-token\") pod \"69467a1c-7315-4154-85a4-72473a1ecd9d\" (UID: \"69467a1c-7315-4154-85a4-72473a1ecd9d\") "
Oct 30 20:42:18.896084 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.895884    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-workload-certs" pod "69467a1c-7315-4154-85a4-72473a1ecd9d" (UID: "69467a1c-7315-4154-85a4-72473a1ecd9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:18.896084 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.896052    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-credential-socket" pod "69467a1c-7315-4154-85a4-72473a1ecd9d" (UID: "69467a1c-7315-4154-85a4-72473a1ecd9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:18.896195 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.896140    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69467a1c-7315-4154-85a4-72473a1ecd9d-istiod-ca-cert" pod "69467a1c-7315-4154-85a4-72473a1ecd9d" (UID: "69467a1c-7315-4154-85a4-72473a1ecd9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:42:18.896480 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.896461    4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-data\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:18.896606 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.896485    4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-workload-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:18.896606 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.896498    4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-workload-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:18.896606 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.896512    4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-credential-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:18.896606 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.896523    4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69467a1c-7315-4154-85a4-72473a1ecd9d-istiod-ca-cert\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:18.898011 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.897988    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-podinfo" pod "69467a1c-7315-4154-85a4-72473a1ecd9d" (UID: "69467a1c-7315-4154-85a4-72473a1ecd9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:42:18.898011 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.897993    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69467a1c-7315-4154-85a4-72473a1ecd9d-kube-api-access-9cb4f" pod "69467a1c-7315-4154-85a4-72473a1ecd9d" (UID: "69467a1c-7315-4154-85a4-72473a1ecd9d"). InnerVolumeSpecName "kube-api-access-9cb4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:18.898526 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.898503    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-token" pod "69467a1c-7315-4154-85a4-72473a1ecd9d" (UID: "69467a1c-7315-4154-85a4-72473a1ecd9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:18.898617 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.898531    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-envoy" pod "69467a1c-7315-4154-85a4-72473a1ecd9d" (UID: "69467a1c-7315-4154-85a4-72473a1ecd9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:18.997901 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.997869    4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-podinfo\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:18.997901 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.997902    4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-token\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:18.998150 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.997917    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cb4f\" (UniqueName: \"kubernetes.io/projected/69467a1c-7315-4154-85a4-72473a1ecd9d-kube-api-access-9cb4f\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:18.998150 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:18.997927    4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69467a1c-7315-4154-85a4-72473a1ecd9d-istio-envoy\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:19.417715 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:19.417678    4471 generic.go:356] "Generic (PLEG): container finished" podID="69467a1c-7315-4154-85a4-72473a1ecd9d" containerID="87c6e80a103f5b3b971b5012e650c4d518458f0f13874bb342bfe470317df0bb" exitCode=0
Oct 30 20:42:19.417882 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:19.417744    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"
Oct 30 20:42:19.417882 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:19.417768    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg" event={"ID":"69467a1c-7315-4154-85a4-72473a1ecd9d","Type":"ContainerDied","Data":"87c6e80a103f5b3b971b5012e650c4d518458f0f13874bb342bfe470317df0bb"}
Oct 30 20:42:19.417882 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:19.417864    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg" event={"ID":"69467a1c-7315-4154-85a4-72473a1ecd9d","Type":"ContainerDied","Data":"a0d22adf94f228263cde559ee64a7113fbc39b181eecc3d8d2f323552451ff03"}
Oct 30 20:42:19.418012 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:19.417895    4471 scope.go:122] "RemoveContainer" containerID="87c6e80a103f5b3b971b5012e650c4d518458f0f13874bb342bfe470317df0bb"
Oct 30 20:42:19.428143 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:19.428126    4471 scope.go:122] "RemoveContainer" containerID="87c6e80a103f5b3b971b5012e650c4d518458f0f13874bb342bfe470317df0bb"
Oct 30 20:42:19.428402 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:42:19.428381    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c6e80a103f5b3b971b5012e650c4d518458f0f13874bb342bfe470317df0bb\": container with ID starting with 87c6e80a103f5b3b971b5012e650c4d518458f0f13874bb342bfe470317df0bb not found: ID does not exist" containerID="87c6e80a103f5b3b971b5012e650c4d518458f0f13874bb342bfe470317df0bb"
Oct 30 20:42:19.428461 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:19.428417    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c6e80a103f5b3b971b5012e650c4d518458f0f13874bb342bfe470317df0bb"} err="failed to get container status \"87c6e80a103f5b3b971b5012e650c4d518458f0f13874bb342bfe470317df0bb\": rpc error: code = NotFound desc = could not find container \"87c6e80a103f5b3b971b5012e650c4d518458f0f13874bb342bfe470317df0bb\": container with ID starting with 87c6e80a103f5b3b971b5012e650c4d518458f0f13874bb342bfe470317df0bb not found: ID does not exist"
Oct 30 20:42:19.601568 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:19.601538    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"]
Oct 30 20:42:19.650973 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:19.650944    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-ctppg"]
Oct 30 20:42:20.431009 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:20.430982    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69467a1c-7315-4154-85a4-72473a1ecd9d" path="/var/lib/kubelet/pods/69467a1c-7315-4154-85a4-72473a1ecd9d/volumes"
Oct 30 20:42:28.747338 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:28.747302    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"]
Oct 30 20:42:28.747794 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:28.747780    4471 cpu_manager.go:414] "containerMap: removing container" podUID="69467a1c-7315-4154-85a4-72473a1ecd9d" containerName="istio-proxy"
Oct 30 20:42:28.747837 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:28.747796    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69467a1c-7315-4154-85a4-72473a1ecd9d" containerName="istio-proxy"
Oct 30 20:42:28.747881 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:28.747873    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="69467a1c-7315-4154-85a4-72473a1ecd9d" containerName="istio-proxy"
Oct 30 20:42:28.752643 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:28.752619    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:28.899768 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:28.899734    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"]
Oct 30 20:42:28.911315 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:28.911286    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:28.911488 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:28.911333    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggx22\" (UniqueName: \"kubernetes.io/projected/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-kube-api-access-ggx22\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:28.911488 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:28.911412    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:28.911488 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:28.911452    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:28.911660 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:28.911511    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:28.911718 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:28.911689    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:28.911775 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:28.911733    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:28.911834 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:28.911818    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:28.911988 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:28.911963    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:29.013701 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.013602    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:29.013701 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.013649    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:29.013701 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.013678    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:29.013994 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.013785    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:29.013994 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.013820    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:29.013994 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.013848    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggx22\" (UniqueName: \"kubernetes.io/projected/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-kube-api-access-ggx22\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:29.013994 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.013879    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:29.013994 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.013907    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:29.013994 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.013982    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:29.014257 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.014073    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:29.014257 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.014178    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:29.014392 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.014366    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:29.014454 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.014402    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:29.014789 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.014765    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:29.016110 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.016088    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:29.016764 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.016736    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:29.073488 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.073457    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggx22\" (UniqueName: \"kubernetes.io/projected/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-kube-api-access-ggx22\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:29.074204 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.074184    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-695mh\" (UID: \"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:29.365697 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.365624    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:29.851214 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.851179    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"]
Oct 30 20:42:29.859045 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:42:29.859017    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2a6fee_e200_4ebc_a47d_2a47a7c3c9cb.slice/crio-3332bf2bd857e8ea72aa1af71de56bbeccdd9ea75bddcbae8582c612cff670e1 WatchSource:0}: Error finding container 3332bf2bd857e8ea72aa1af71de56bbeccdd9ea75bddcbae8582c612cff670e1: Status 404 returned error can't find the container with id 3332bf2bd857e8ea72aa1af71de56bbeccdd9ea75bddcbae8582c612cff670e1
Oct 30 20:42:29.861521 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.861481    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:42:29.861612 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.861558    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:42:29.861612 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:29.861595    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:42:30.480034 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:30.479994    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh" event={"ID":"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb","Type":"ContainerStarted","Data":"6c302546e61d37c34d9118c75c8ba10a32bef7b0413e74d2a766ff5d685ff037"}
Oct 30 20:42:30.480252 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:30.480058    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh" event={"ID":"fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb","Type":"ContainerStarted","Data":"3332bf2bd857e8ea72aa1af71de56bbeccdd9ea75bddcbae8582c612cff670e1"}
Oct 30 20:42:30.699853 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:30.699797    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh" podStartSLOduration=2.699783181 podStartE2EDuration="2.699783181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:30.649280281 +0000 UTC m=+1370.327908900" watchObservedRunningTime="2026-10-30 20:42:30.699783181 +0000 UTC m=+1370.378411796"
Oct 30 20:42:30.979838 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:42:30.979804    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:42:31.367494 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:31.367393    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:31.368449 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:31.368423    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-695mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 20:42:31.368594 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:31.368481    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh" podUID="fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 20:42:32.367790 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:32.367746    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-695mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 20:42:32.368194 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:32.367818    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh" podUID="fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 20:42:33.367881 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:33.367840    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-695mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 20:42:33.368354 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:33.367910    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh" podUID="fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 20:42:34.367349 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:34.366697    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-695mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 20:42:34.367349 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:34.366767    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh" podUID="fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 20:42:35.366718 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:35.366631    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-695mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 20:42:35.367305 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:35.367275    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh" podUID="fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 20:42:36.127749 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:36.127713    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"]
Oct 30 20:42:36.131764 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:36.131742    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"
Oct 30 20:42:36.173259 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:36.173216    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcqhc\" (UniqueName: \"kubernetes.io/projected/33e13649-3830-4327-a755-e0693a62bc71-kube-api-access-fcqhc\") pod \"maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh\" (UID: \"33e13649-3830-4327-a755-e0693a62bc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"
Oct 30 20:42:36.173453 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:36.173270    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e13649-3830-4327-a755-e0693a62bc71-openshift-service-ca\") pod \"maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh\" (UID: \"33e13649-3830-4327-a755-e0693a62bc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"
Oct 30 20:42:36.173453 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:36.173292    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e13649-3830-4327-a755-e0693a62bc71-maas-api-tls\") pod \"maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh\" (UID: \"33e13649-3830-4327-a755-e0693a62bc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"
Oct 30 20:42:36.173533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:36.173456    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e13649-3830-4327-a755-e0693a62bc71-metrics-certs\") pod \"maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh\" (UID: \"33e13649-3830-4327-a755-e0693a62bc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"
Oct 30 20:42:36.185428 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:36.185406    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8660b337-serving-cert\""
Oct 30 20:42:36.236159 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:36.236130    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"]
Oct 30 20:42:36.275087 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:36.274414    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e13649-3830-4327-a755-e0693a62bc71-openshift-service-ca\") pod \"maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh\" (UID: \"33e13649-3830-4327-a755-e0693a62bc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"
Oct 30 20:42:36.275087 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:36.274468    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e13649-3830-4327-a755-e0693a62bc71-maas-api-tls\") pod \"maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh\" (UID: \"33e13649-3830-4327-a755-e0693a62bc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"
Oct 30 20:42:36.275087 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:36.274522    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e13649-3830-4327-a755-e0693a62bc71-metrics-certs\") pod \"maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh\" (UID: \"33e13649-3830-4327-a755-e0693a62bc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"
Oct 30 20:42:36.275087 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:36.274709    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcqhc\" (UniqueName: \"kubernetes.io/projected/33e13649-3830-4327-a755-e0693a62bc71-kube-api-access-fcqhc\") pod \"maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh\" (UID: \"33e13649-3830-4327-a755-e0693a62bc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"
Oct 30 20:42:36.275610 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:42:36.275588    4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-serving-cert: secret "maas-api-e2e-ait-8660b337-serving-cert" not found
Oct 30 20:42:36.275794 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:42:36.275779    4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33e13649-3830-4327-a755-e0693a62bc71-maas-api-tls podName:33e13649-3830-4327-a755-e0693a62bc71 nodeName:}" failed. No retries permitted until 2026-10-30 20:42:36.775758598 +0000 UTC m=+1376.454387209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33e13649-3830-4327-a755-e0693a62bc71-maas-api-tls") pod "maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh" (UID: "33e13649-3830-4327-a755-e0693a62bc71") : secret "maas-api-e2e-ait-8660b337-serving-cert" not found
Oct 30 20:42:36.275903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:36.275785    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e13649-3830-4327-a755-e0693a62bc71-openshift-service-ca\") pod \"maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh\" (UID: \"33e13649-3830-4327-a755-e0693a62bc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"
Oct 30 20:42:36.279022 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:36.278998    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e13649-3830-4327-a755-e0693a62bc71-metrics-certs\") pod \"maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh\" (UID: \"33e13649-3830-4327-a755-e0693a62bc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"
Oct 30 20:42:36.332085 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:36.332061    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcqhc\" (UniqueName: \"kubernetes.io/projected/33e13649-3830-4327-a755-e0693a62bc71-kube-api-access-fcqhc\") pod \"maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh\" (UID: \"33e13649-3830-4327-a755-e0693a62bc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"
Oct 30 20:42:36.369806 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:36.369770    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-695mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 20:42:36.370491 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:36.370458    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh" podUID="fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 20:42:36.785948 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:36.785902    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e13649-3830-4327-a755-e0693a62bc71-maas-api-tls\") pod \"maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh\" (UID: \"33e13649-3830-4327-a755-e0693a62bc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"
Oct 30 20:42:36.788408 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:36.788382    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e13649-3830-4327-a755-e0693a62bc71-maas-api-tls\") pod \"maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh\" (UID: \"33e13649-3830-4327-a755-e0693a62bc71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"
Oct 30 20:42:37.048828 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:37.048747    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"
Oct 30 20:42:37.366656 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:37.366561    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-695mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 20:42:37.366656 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:37.366628    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh" podUID="fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 20:42:37.430615 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:37.430585    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"]
Oct 30 20:42:37.431621 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:42:37.431596    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e13649_3830_4327_a755_e0693a62bc71.slice/crio-d7c695c6a6a721a09b5630771d70c768f787c2d6a1271d4cbc5006c2f237b5c6 WatchSource:0}: Error finding container d7c695c6a6a721a09b5630771d70c768f787c2d6a1271d4cbc5006c2f237b5c6: Status 404 returned error can't find the container with id d7c695c6a6a721a09b5630771d70c768f787c2d6a1271d4cbc5006c2f237b5c6
Oct 30 20:42:37.559624 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:37.559584    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh" event={"ID":"33e13649-3830-4327-a755-e0693a62bc71","Type":"ContainerStarted","Data":"d7c695c6a6a721a09b5630771d70c768f787c2d6a1271d4cbc5006c2f237b5c6"}
Oct 30 20:42:39.367707 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:39.367655    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-695mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:42:39.368234 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:39.367738    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh" podUID="fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:39.388180 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:39.388146    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:39.388545 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:39.388509    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:39.389468 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:39.389443    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh"
Oct 30 20:42:39.621965 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:39.617727    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh" event={"ID":"33e13649-3830-4327-a755-e0693a62bc71","Type":"ContainerStarted","Data":"8dcceef76388aaa301de2ffe73d7c6bc5324e7c29aca93a300bb1d4180abcaf3"}
Oct 30 20:42:39.621965 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:39.617973    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"
Oct 30 20:42:39.811542 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:39.811475    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh" podStartSLOduration=2.819063339 podStartE2EDuration="3.811458043s" totalImagesPullingTime="992.394704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:39.805898895 +0000 UTC m=+1379.484527512" watchObservedRunningTime="2026-10-30 20:42:39.811458043 +0000 UTC m=+1379.490086659"
Oct 30 20:42:43.838037 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:43.838005    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"]
Oct 30 20:42:43.843199 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:43.843178    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:43.925096 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:43.925055    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:43.925256 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:43.925168    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:43.925256 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:43.925204    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:43.925369 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:43.925264    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:43.925418 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:43.925377    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:43.925451 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:43.925419    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:43.925537 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:43.925520    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:43.925681 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:43.925660    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlcmq\" (UniqueName: \"kubernetes.io/projected/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-kube-api-access-nlcmq\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:43.925727 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:43.925716    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:44.026667 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.026638    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlcmq\" (UniqueName: \"kubernetes.io/projected/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-kube-api-access-nlcmq\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:44.026828 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.026683    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:44.026888 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.026866    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:44.026961 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.026945    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:44.027024 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.026976    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:44.027024 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.027008    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:44.027129 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.027059    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:44.027129 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.027096    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:44.027246 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.027164    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:44.027246 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.027212    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:44.027465 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.027442    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:44.027588 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.027562    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:44.027658 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.027604    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:44.027857 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.027833    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:44.029445 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.029414    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:44.030117 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.030094    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:44.083932 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.083902    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlcmq\" (UniqueName: \"kubernetes.io/projected/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-kube-api-access-nlcmq\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:44.084255 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.084233    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ac0a5d8-9d3d-4a31-ac67-16c610d242ff-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-7g5ll\" (UID: \"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:44.096884 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.096775    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"]
Oct 30 20:42:44.156032 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.156000    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:44.595970 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.595946    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"]
Oct 30 20:42:44.597518 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:42:44.597491    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac0a5d8_9d3d_4a31_ac67_16c610d242ff.slice/crio-2f0d92d6860550ae2111781e5d9edf01942b4ccc3ba01e64350f64da9eaa257a WatchSource:0}: Error finding container 2f0d92d6860550ae2111781e5d9edf01942b4ccc3ba01e64350f64da9eaa257a: Status 404 returned error can't find the container with id 2f0d92d6860550ae2111781e5d9edf01942b4ccc3ba01e64350f64da9eaa257a
Oct 30 20:42:44.599951 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.599920    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:42:44.600035 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.599983    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:42:44.600035 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.600012    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:42:44.820424 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.820380    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll" event={"ID":"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff","Type":"ContainerStarted","Data":"973e66c6f580d69b7143e360417177a38b3db8a84770018421439cfc47916957"}
Oct 30 20:42:44.820577 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.820444    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll" event={"ID":"1ac0a5d8-9d3d-4a31-ac67-16c610d242ff","Type":"ContainerStarted","Data":"2f0d92d6860550ae2111781e5d9edf01942b4ccc3ba01e64350f64da9eaa257a"}
Oct 30 20:42:44.943532 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:44.943291    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll" podStartSLOduration=1.943274203 podStartE2EDuration="1.943274203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:44.942418067 +0000 UTC m=+1384.621046683" watchObservedRunningTime="2026-10-30 20:42:44.943274203 +0000 UTC m=+1384.621902831"
Oct 30 20:42:45.162346 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:45.159687    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:45.167500 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:45.167457    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7g5ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 20:42:45.167767 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:45.167742    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll" podUID="1ac0a5d8-9d3d-4a31-ac67-16c610d242ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 20:42:46.158725 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:46.158686    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7g5ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 20:42:46.159402 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:46.159371    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll" podUID="1ac0a5d8-9d3d-4a31-ac67-16c610d242ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 20:42:46.713936 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:46.713905    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"]
Oct 30 20:42:46.717805 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:46.717783    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"
Oct 30 20:42:46.769709 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:46.769675    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-23cde229-serving-cert\""
Oct 30 20:42:46.781658 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:46.781624    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-metrics-certs\") pod \"maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4\" (UID: \"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"
Oct 30 20:42:46.781820 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:46.781756    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-maas-api-tls\") pod \"maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4\" (UID: \"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"
Oct 30 20:42:46.781820 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:46.781792    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghmps\" (UniqueName: \"kubernetes.io/projected/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-kube-api-access-ghmps\") pod \"maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4\" (UID: \"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"
Oct 30 20:42:46.782124 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:46.782097    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-openshift-service-ca\") pod \"maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4\" (UID: \"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"
Oct 30 20:42:46.783832 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:46.783804    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"]
Oct 30 20:42:46.801302 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:46.801281    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"
Oct 30 20:42:46.884299 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:46.884257    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-openshift-service-ca\") pod \"maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4\" (UID: \"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"
Oct 30 20:42:46.884514 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:46.884351    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-metrics-certs\") pod \"maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4\" (UID: \"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"
Oct 30 20:42:46.884514 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:46.884431    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-maas-api-tls\") pod \"maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4\" (UID: \"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"
Oct 30 20:42:46.884514 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:46.884463    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmps\" (UniqueName: \"kubernetes.io/projected/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-kube-api-access-ghmps\") pod \"maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4\" (UID: \"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"
Oct 30 20:42:46.885308 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:42:46.885281    4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-serving-cert: secret "maas-api-e2e-ait-23cde229-serving-cert" not found
Oct 30 20:42:46.885453 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:42:46.885390    4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-maas-api-tls podName:f58aaf1a-16c8-400e-8c20-20bf9d73ff4a nodeName:}" failed. No retries permitted until 2026-10-30 20:42:47.385365905 +0000 UTC m=+1387.063994512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-maas-api-tls") pod "maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4" (UID: "f58aaf1a-16c8-400e-8c20-20bf9d73ff4a") : secret "maas-api-e2e-ait-23cde229-serving-cert" not found
Oct 30 20:42:46.886178 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:46.886149    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-openshift-service-ca\") pod \"maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4\" (UID: \"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"
Oct 30 20:42:46.887403 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:46.887380    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-metrics-certs\") pod \"maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4\" (UID: \"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"
Oct 30 20:42:46.989379 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:46.989284    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghmps\" (UniqueName: \"kubernetes.io/projected/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-kube-api-access-ghmps\") pod \"maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4\" (UID: \"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"
Oct 30 20:42:47.162342 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:47.161413    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7g5ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 20:42:47.162342 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:47.161495    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll" podUID="1ac0a5d8-9d3d-4a31-ac67-16c610d242ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 20:42:47.469107 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:47.469068    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-maas-api-tls\") pod \"maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4\" (UID: \"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"
Oct 30 20:42:47.471707 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:47.471684    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-maas-api-tls\") pod \"maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4\" (UID: \"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"
Oct 30 20:42:47.631166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:47.631136    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"
Oct 30 20:42:48.038615 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:48.038589    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"]
Oct 30 20:42:48.040552 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:42:48.040527    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58aaf1a_16c8_400e_8c20_20bf9d73ff4a.slice/crio-67a66669a7fbe6a5597f1fa16a572ad8b4f99db7c22eb3fa5c4f6835b0e19a0f WatchSource:0}: Error finding container 67a66669a7fbe6a5597f1fa16a572ad8b4f99db7c22eb3fa5c4f6835b0e19a0f: Status 404 returned error can't find the container with id 67a66669a7fbe6a5597f1fa16a572ad8b4f99db7c22eb3fa5c4f6835b0e19a0f
Oct 30 20:42:48.163458 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:48.163423    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7g5ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 20:42:48.164032 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:48.163999    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll" podUID="1ac0a5d8-9d3d-4a31-ac67-16c610d242ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 20:42:48.880736 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:48.880695    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4" event={"ID":"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a","Type":"ContainerStarted","Data":"67a66669a7fbe6a5597f1fa16a572ad8b4f99db7c22eb3fa5c4f6835b0e19a0f"}
Oct 30 20:42:49.164358 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:49.163495    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7g5ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 20:42:49.164358 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:49.163551    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll" podUID="1ac0a5d8-9d3d-4a31-ac67-16c610d242ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 20:42:49.978765 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:49.978719    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4" event={"ID":"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a","Type":"ContainerStarted","Data":"ad8c70d752138c4a4c9bc911c2aa34772e863c5276890a4ba12c7624f9acf2a8"}
Oct 30 20:42:49.980118 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:49.980091    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"
Oct 30 20:42:50.102745 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:50.102688    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4" podStartSLOduration=3.043749082 podStartE2EDuration="4.102672595s" totalImagesPullingTime="1.058923513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:50.099916599 +0000 UTC m=+1389.778545215" watchObservedRunningTime="2026-10-30 20:42:50.102672595 +0000 UTC m=+1389.781301244"
Oct 30 20:42:50.164768 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:50.164725    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7g5ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 20:42:50.165438 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:50.165404    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll" podUID="1ac0a5d8-9d3d-4a31-ac67-16c610d242ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 20:42:51.165560 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:51.158793    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7g5ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 20:42:51.165560 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:51.158875    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll" podUID="1ac0a5d8-9d3d-4a31-ac67-16c610d242ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 20:42:52.156407 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.156308    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7g5ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 20:42:52.156407 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.156392    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll" podUID="1ac0a5d8-9d3d-4a31-ac67-16c610d242ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 20:42:52.624675 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.624648    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"]
Oct 30 20:42:52.629018 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.628992    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.665223 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.665197    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c120141b-7364-4435-9533-046a104f988b-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.665396 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.665238    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.665452 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.665394    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c120141b-7364-4435-9533-046a104f988b-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.665558 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.665541    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktrzd\" (UniqueName: \"kubernetes.io/projected/c120141b-7364-4435-9533-046a104f988b-kube-api-access-ktrzd\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.665613 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.665570    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.665613 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.665590    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.665718 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.665699    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.665777 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.665732    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c120141b-7364-4435-9533-046a104f988b-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.665919 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.665900    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.767665 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.767622    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.767833 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.767673    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.767833 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.767717    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.767833 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.767743    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c120141b-7364-4435-9533-046a104f988b-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.768242 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.767858    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.768242 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.767906    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c120141b-7364-4435-9533-046a104f988b-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.768242 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.767951    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.768242 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.768031    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c120141b-7364-4435-9533-046a104f988b-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.768242 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.768141    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktrzd\" (UniqueName: \"kubernetes.io/projected/c120141b-7364-4435-9533-046a104f988b-kube-api-access-ktrzd\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.768242 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.768140    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-workload-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.769442 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.769032    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c120141b-7364-4435-9533-046a104f988b-istiod-ca-cert\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.769582 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.769161    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-workload-certs\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.769582 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.769412    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-credential-socket\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.770070 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.770044    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-istio-data\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.772899 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.772871    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c120141b-7364-4435-9533-046a104f988b-istio-podinfo\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.773496 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.773472    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-istio-envoy\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.830445 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.830385    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktrzd\" (UniqueName: \"kubernetes.io/projected/c120141b-7364-4435-9533-046a104f988b-kube-api-access-ktrzd\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.831802 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.831772    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"]
Oct 30 20:42:52.832066 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.832044    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c120141b-7364-4435-9533-046a104f988b-istio-token\") pod \"e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") " pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:52.944498 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:52.944412    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:53.272286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:53.272199    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:53.272929 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:53.272889    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:53.273712 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:53.273692    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll"
Oct 30 20:42:53.391170 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:53.391143    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"]
Oct 30 20:42:53.392861 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:42:53.392838    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc120141b_7364_4435_9533_046a104f988b.slice/crio-f28f988eb6a8c3c7d967c089cf689610e687f8058694354ac59982c400150164 WatchSource:0}: Error finding container f28f988eb6a8c3c7d967c089cf689610e687f8058694354ac59982c400150164: Status 404 returned error can't find the container with id f28f988eb6a8c3c7d967c089cf689610e687f8058694354ac59982c400150164
Oct 30 20:42:53.395278 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:53.395242    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:42:53.395420 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:53.395316    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:42:53.395420 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:53.395401    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:42:54.012937 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:54.012904    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd" event={"ID":"c120141b-7364-4435-9533-046a104f988b","Type":"ContainerStarted","Data":"7a724c54f15da025a926991823bc597fca8fe71d721271b12bf432d83bdc5da2"}
Oct 30 20:42:54.013313 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:54.012981    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd" event={"ID":"c120141b-7364-4435-9533-046a104f988b","Type":"ContainerStarted","Data":"f28f988eb6a8c3c7d967c089cf689610e687f8058694354ac59982c400150164"}
Oct 30 20:42:54.134886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:54.134825    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd" podStartSLOduration=2.134809768 podStartE2EDuration="2.134809768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:54.134422197 +0000 UTC m=+1393.813050815" watchObservedRunningTime="2026-10-30 20:42:54.134809768 +0000 UTC m=+1393.813438383"
Oct 30 20:42:54.957116 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:54.949760    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:42:54.967425 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:54.962469    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 20:42:54.967425 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:54.962537    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd" podUID="c120141b-7364-4435-9533-046a104f988b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 20:42:55.958347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:55.946726    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 20:42:55.958347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:55.946788    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd" podUID="c120141b-7364-4435-9533-046a104f988b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 20:42:56.964348 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:56.947706    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 20:42:56.964348 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:56.947766    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd" podUID="c120141b-7364-4435-9533-046a104f988b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 20:42:56.996471 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:56.996437    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"
Oct 30 20:42:57.950827 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:57.950782    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 20:42:57.951175 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:57.951141    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd" podUID="c120141b-7364-4435-9533-046a104f988b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 20:42:58.960357 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:58.954687    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 20:42:58.960357 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:58.954763    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd" podUID="c120141b-7364-4435-9533-046a104f988b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 20:42:59.947717 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:59.947678    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 20:42:59.948139 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:42:59.948106    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd" podUID="c120141b-7364-4435-9533-046a104f988b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 20:43:00.945152 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:43:00.945117    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 20:43:00.945547 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:43:00.945202    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd" podUID="c120141b-7364-4435-9533-046a104f988b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 20:43:00.956213 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:43:00.956179    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:43:01.949524 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:43:01.949494    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:43:01.950006 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:43:01.949967    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:43:01.950767 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:43:01.950747    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 20:43:30.954802 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:43:30.954765    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:44:01.043862 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:44:01.043818    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:44:20.390979 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:20.390939    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"]
Oct 30 20:44:20.393482 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:20.391705    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh" podUID="33e13649-3830-4327-a755-e0693a62bc71" containerName="maas-api" containerID="cri-o://8dcceef76388aaa301de2ffe73d7c6bc5324e7c29aca93a300bb1d4180abcaf3" gracePeriod=30
Oct 30 20:44:20.839412 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:20.835505    4471 generic.go:356] "Generic (PLEG): container finished" podID="33e13649-3830-4327-a755-e0693a62bc71" containerID="8dcceef76388aaa301de2ffe73d7c6bc5324e7c29aca93a300bb1d4180abcaf3" exitCode=0
Oct 30 20:44:20.839412 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:20.835733    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh" event={"ID":"33e13649-3830-4327-a755-e0693a62bc71","Type":"ContainerDied","Data":"8dcceef76388aaa301de2ffe73d7c6bc5324e7c29aca93a300bb1d4180abcaf3"}
Oct 30 20:44:21.037973 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:21.037949    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"
Oct 30 20:44:21.121353 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:21.121231    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33e13649-3830-4327-a755-e0693a62bc71-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e13649-3830-4327-a755-e0693a62bc71-maas-api-tls\") pod \"33e13649-3830-4327-a755-e0693a62bc71\" (UID: \"33e13649-3830-4327-a755-e0693a62bc71\") "
Oct 30 20:44:21.121353 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:21.121274    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e13649-3830-4327-a755-e0693a62bc71-kube-api-access-fcqhc\" (UniqueName: \"kubernetes.io/projected/33e13649-3830-4327-a755-e0693a62bc71-kube-api-access-fcqhc\") pod \"33e13649-3830-4327-a755-e0693a62bc71\" (UID: \"33e13649-3830-4327-a755-e0693a62bc71\") "
Oct 30 20:44:21.121566 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:21.121357    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e13649-3830-4327-a755-e0693a62bc71-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e13649-3830-4327-a755-e0693a62bc71-openshift-service-ca\") pod \"33e13649-3830-4327-a755-e0693a62bc71\" (UID: \"33e13649-3830-4327-a755-e0693a62bc71\") "
Oct 30 20:44:21.121566 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:21.121396    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33e13649-3830-4327-a755-e0693a62bc71-metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e13649-3830-4327-a755-e0693a62bc71-metrics-certs\") pod \"33e13649-3830-4327-a755-e0693a62bc71\" (UID: \"33e13649-3830-4327-a755-e0693a62bc71\") "
Oct 30 20:44:21.122427 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:21.122394    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e13649-3830-4327-a755-e0693a62bc71-openshift-service-ca" pod "33e13649-3830-4327-a755-e0693a62bc71" (UID: "33e13649-3830-4327-a755-e0693a62bc71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:44:21.124173 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:21.124149    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e13649-3830-4327-a755-e0693a62bc71-metrics-certs" pod "33e13649-3830-4327-a755-e0693a62bc71" (UID: "33e13649-3830-4327-a755-e0693a62bc71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:44:21.124416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:21.124392    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e13649-3830-4327-a755-e0693a62bc71-kube-api-access-fcqhc" pod "33e13649-3830-4327-a755-e0693a62bc71" (UID: "33e13649-3830-4327-a755-e0693a62bc71"). InnerVolumeSpecName "kube-api-access-fcqhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:21.124416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:21.124401    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e13649-3830-4327-a755-e0693a62bc71-maas-api-tls" pod "33e13649-3830-4327-a755-e0693a62bc71" (UID: "33e13649-3830-4327-a755-e0693a62bc71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:44:21.223171 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:21.223144    4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e13649-3830-4327-a755-e0693a62bc71-maas-api-tls\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:21.223171 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:21.223170    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcqhc\" (UniqueName: \"kubernetes.io/projected/33e13649-3830-4327-a755-e0693a62bc71-kube-api-access-fcqhc\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:21.223394 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:21.223202    4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e13649-3830-4327-a755-e0693a62bc71-openshift-service-ca\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:21.223394 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:21.223212    4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e13649-3830-4327-a755-e0693a62bc71-metrics-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:21.871890 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:21.871851    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh" event={"ID":"33e13649-3830-4327-a755-e0693a62bc71","Type":"ContainerDied","Data":"d7c695c6a6a721a09b5630771d70c768f787c2d6a1271d4cbc5006c2f237b5c6"}
Oct 30 20:44:21.872639 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:21.872619    4471 scope.go:122] "RemoveContainer" containerID="8dcceef76388aaa301de2ffe73d7c6bc5324e7c29aca93a300bb1d4180abcaf3"
Oct 30 20:44:21.872943 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:21.872928    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"
Oct 30 20:44:22.061391 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:22.061273    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"]
Oct 30 20:44:22.063945 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:22.063920    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8660b337-5bc79b6944-rcqbh"]
Oct 30 20:44:22.431446 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:22.431412    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e13649-3830-4327-a755-e0693a62bc71" path="/var/lib/kubelet/pods/33e13649-3830-4327-a755-e0693a62bc71/volumes"
Oct 30 20:44:27.597679 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:27.597644    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"]
Oct 30 20:44:27.598093 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:27.597849    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4" podUID="f58aaf1a-16c8-400e-8c20-20bf9d73ff4a" containerName="maas-api" containerID="cri-o://ad8c70d752138c4a4c9bc911c2aa34772e863c5276890a4ba12c7624f9acf2a8" gracePeriod=30
Oct 30 20:44:28.094955 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:28.094916    4471 generic.go:356] "Generic (PLEG): container finished" podID="f58aaf1a-16c8-400e-8c20-20bf9d73ff4a" containerID="ad8c70d752138c4a4c9bc911c2aa34772e863c5276890a4ba12c7624f9acf2a8" exitCode=0
Oct 30 20:44:28.095388 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:28.095361    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4" event={"ID":"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a","Type":"ContainerDied","Data":"ad8c70d752138c4a4c9bc911c2aa34772e863c5276890a4ba12c7624f9acf2a8"}
Oct 30 20:44:28.204287 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:28.204256    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"
Oct 30 20:44:28.302445 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:28.302410    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-maas-api-tls\") pod \"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a\" (UID: \"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a\") "
Oct 30 20:44:28.302707 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:28.302504    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-openshift-service-ca\") pod \"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a\" (UID: \"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a\") "
Oct 30 20:44:28.302707 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:28.302556    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-kube-api-access-ghmps\" (UniqueName: \"kubernetes.io/projected/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-kube-api-access-ghmps\") pod \"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a\" (UID: \"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a\") "
Oct 30 20:44:28.302707 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:28.302644    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-metrics-certs\") pod \"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a\" (UID: \"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a\") "
Oct 30 20:44:28.303501 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:28.303454    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-openshift-service-ca" pod "f58aaf1a-16c8-400e-8c20-20bf9d73ff4a" (UID: "f58aaf1a-16c8-400e-8c20-20bf9d73ff4a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:44:28.306002 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:28.305979    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-kube-api-access-ghmps" pod "f58aaf1a-16c8-400e-8c20-20bf9d73ff4a" (UID: "f58aaf1a-16c8-400e-8c20-20bf9d73ff4a"). InnerVolumeSpecName "kube-api-access-ghmps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:28.306553 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:28.306528    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-metrics-certs" pod "f58aaf1a-16c8-400e-8c20-20bf9d73ff4a" (UID: "f58aaf1a-16c8-400e-8c20-20bf9d73ff4a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:44:28.306785 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:28.306756    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-maas-api-tls" pod "f58aaf1a-16c8-400e-8c20-20bf9d73ff4a" (UID: "f58aaf1a-16c8-400e-8c20-20bf9d73ff4a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:44:28.404616 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:28.404579    4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-metrics-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:28.404616 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:28.404610    4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-maas-api-tls\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:28.404616 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:28.404621    4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-openshift-service-ca\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:28.404867 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:28.404630    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghmps\" (UniqueName: \"kubernetes.io/projected/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a-kube-api-access-ghmps\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:29.102765 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:29.102724    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4" event={"ID":"f58aaf1a-16c8-400e-8c20-20bf9d73ff4a","Type":"ContainerDied","Data":"67a66669a7fbe6a5597f1fa16a572ad8b4f99db7c22eb3fa5c4f6835b0e19a0f"}
Oct 30 20:44:29.103381 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:29.102740    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"
Oct 30 20:44:29.103490 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:29.103379    4471 scope.go:122] "RemoveContainer" containerID="ad8c70d752138c4a4c9bc911c2aa34772e863c5276890a4ba12c7624f9acf2a8"
Oct 30 20:44:29.267204 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:29.267168    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"]
Oct 30 20:44:29.317388 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:29.317353    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23cde229-59d95fc8f9-blhq4"]
Oct 30 20:44:30.443156 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:30.443116    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58aaf1a-16c8-400e-8c20-20bf9d73ff4a" path="/var/lib/kubelet/pods/f58aaf1a-16c8-400e-8c20-20bf9d73ff4a/volumes"
Oct 30 20:44:30.955614 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:44:30.955574    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:44:35.358593 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.358498    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h"]
Oct 30 20:44:35.360517 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.359906    4471 cpu_manager.go:414] "containerMap: removing container" podUID="33e13649-3830-4327-a755-e0693a62bc71" containerName="maas-api"
Oct 30 20:44:35.360736 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.360714    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e13649-3830-4327-a755-e0693a62bc71" containerName="maas-api"
Oct 30 20:44:35.360914 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.360899    4471 cpu_manager.go:414] "containerMap: removing container" podUID="f58aaf1a-16c8-400e-8c20-20bf9d73ff4a" containerName="maas-api"
Oct 30 20:44:35.361011 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.361000    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58aaf1a-16c8-400e-8c20-20bf9d73ff4a" containerName="maas-api"
Oct 30 20:44:35.361281 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.361268    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e13649-3830-4327-a755-e0693a62bc71" containerName="maas-api"
Oct 30 20:44:35.361415 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.361405    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58aaf1a-16c8-400e-8c20-20bf9d73ff4a" containerName="maas-api"
Oct 30 20:44:35.369662 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.369638    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h"
Oct 30 20:44:35.415380 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.415343    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h"]
Oct 30 20:44:35.480343 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.466661    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-c2bdb58a-dockercfg-wcvtd\""
Oct 30 20:44:35.480343 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.467548    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-c2bdb58a\""
Oct 30 20:44:35.526846 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.526808    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf"]
Oct 30 20:44:35.587392 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.569979    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf"
Oct 30 20:44:35.629356 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.627830    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e1cfa20-1a50-46be-9967-cd4c6f8075bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h\" (UID: \"3e1cfa20-1a50-46be-9967-cd4c6f8075bc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h"
Oct 30 20:44:35.629356 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.628126    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsd88\" (UniqueName: \"kubernetes.io/projected/3e1cfa20-1a50-46be-9967-cd4c6f8075bc-kube-api-access-qsd88\") pod \"payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h\" (UID: \"3e1cfa20-1a50-46be-9967-cd4c6f8075bc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h"
Oct 30 20:44:35.672486 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.662846    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf"]
Oct 30 20:44:35.729623 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.729585    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsd88\" (UniqueName: \"kubernetes.io/projected/3e1cfa20-1a50-46be-9967-cd4c6f8075bc-kube-api-access-qsd88\") pod \"payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h\" (UID: \"3e1cfa20-1a50-46be-9967-cd4c6f8075bc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h"
Oct 30 20:44:35.729803 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.729666    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e1cfa20-1a50-46be-9967-cd4c6f8075bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h\" (UID: \"3e1cfa20-1a50-46be-9967-cd4c6f8075bc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h"
Oct 30 20:44:35.729803 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.729779    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrjcm\" (UniqueName: \"kubernetes.io/projected/e6178eb2-618b-4e28-bff5-86952e4965a7-kube-api-access-lrjcm\") pod \"payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf\" (UID: \"e6178eb2-618b-4e28-bff5-86952e4965a7\") " pod="openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf"
Oct 30 20:44:35.729914 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.729828    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6178eb2-618b-4e28-bff5-86952e4965a7-plugins-config-volume\") pod \"payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf\" (UID: \"e6178eb2-618b-4e28-bff5-86952e4965a7\") " pod="openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf"
Oct 30 20:44:35.730446 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.730426    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e1cfa20-1a50-46be-9967-cd4c6f8075bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h\" (UID: \"3e1cfa20-1a50-46be-9967-cd4c6f8075bc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h"
Oct 30 20:44:35.786247 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.786216    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsd88\" (UniqueName: \"kubernetes.io/projected/3e1cfa20-1a50-46be-9967-cd4c6f8075bc-kube-api-access-qsd88\") pod \"payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h\" (UID: \"3e1cfa20-1a50-46be-9967-cd4c6f8075bc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h"
Oct 30 20:44:35.830810 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.830776    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrjcm\" (UniqueName: \"kubernetes.io/projected/e6178eb2-618b-4e28-bff5-86952e4965a7-kube-api-access-lrjcm\") pod \"payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf\" (UID: \"e6178eb2-618b-4e28-bff5-86952e4965a7\") " pod="openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf"
Oct 30 20:44:35.830975 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.830846    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6178eb2-618b-4e28-bff5-86952e4965a7-plugins-config-volume\") pod \"payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf\" (UID: \"e6178eb2-618b-4e28-bff5-86952e4965a7\") " pod="openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf"
Oct 30 20:44:35.831610 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.831584    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6178eb2-618b-4e28-bff5-86952e4965a7-plugins-config-volume\") pod \"payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf\" (UID: \"e6178eb2-618b-4e28-bff5-86952e4965a7\") " pod="openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf"
Oct 30 20:44:35.887895 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.887825    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrjcm\" (UniqueName: \"kubernetes.io/projected/e6178eb2-618b-4e28-bff5-86952e4965a7-kube-api-access-lrjcm\") pod \"payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf\" (UID: \"e6178eb2-618b-4e28-bff5-86952e4965a7\") " pod="openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf"
Oct 30 20:44:35.983757 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.983723    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf"
Oct 30 20:44:35.990760 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:35.990741    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h"
Oct 30 20:44:36.367964 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:36.367941    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf"]
Oct 30 20:44:36.369194 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:44:36.369166    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6178eb2_618b_4e28_bff5_86952e4965a7.slice/crio-d4ee0ada23935a5c71c5bbe3313c8317b78438e1e6f86f60943019459032dc0d WatchSource:0}: Error finding container d4ee0ada23935a5c71c5bbe3313c8317b78438e1e6f86f60943019459032dc0d: Status 404 returned error can't find the container with id d4ee0ada23935a5c71c5bbe3313c8317b78438e1e6f86f60943019459032dc0d
Oct 30 20:44:36.416729 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:36.416705    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h"]
Oct 30 20:44:36.417763 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:44:36.417734    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e1cfa20_1a50_46be_9967_cd4c6f8075bc.slice/crio-dba08dd4dd62946df144e7b975da4a6ab6f2aa8a82c043aa49016f8db4673d91 WatchSource:0}: Error finding container dba08dd4dd62946df144e7b975da4a6ab6f2aa8a82c043aa49016f8db4673d91: Status 404 returned error can't find the container with id dba08dd4dd62946df144e7b975da4a6ab6f2aa8a82c043aa49016f8db4673d91
Oct 30 20:44:37.175195 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:37.175145    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf" event={"ID":"e6178eb2-618b-4e28-bff5-86952e4965a7","Type":"ContainerStarted","Data":"d4ee0ada23935a5c71c5bbe3313c8317b78438e1e6f86f60943019459032dc0d"}
Oct 30 20:44:37.176533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:37.176501    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h" event={"ID":"3e1cfa20-1a50-46be-9967-cd4c6f8075bc","Type":"ContainerStarted","Data":"dba08dd4dd62946df144e7b975da4a6ab6f2aa8a82c043aa49016f8db4673d91"}
Oct 30 20:44:40.198220 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:40.198176    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h" event={"ID":"3e1cfa20-1a50-46be-9967-cd4c6f8075bc","Type":"ContainerStarted","Data":"8a59b0fb4ab3a948c6ba83bc2aaac7b6fe80cb56c32980259e1122d9d2f18954"}
Oct 30 20:44:40.198706 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:40.198377    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h"
Oct 30 20:44:40.200057 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:40.200033    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf" event={"ID":"e6178eb2-618b-4e28-bff5-86952e4965a7","Type":"ContainerStarted","Data":"6c3e4313f643d6801f6de39f2a87f57c24b98e2967117608fed6901d643db1ec"}
Oct 30 20:44:40.200188 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:40.200173    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf"
Oct 30 20:44:40.315589 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:40.315539    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h" podStartSLOduration=2.053334667 podStartE2EDuration="5.315523987s" totalImagesPullingTime="3.26218932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:40.312483033 +0000 UTC m=+1499.991111651" watchObservedRunningTime="2026-10-30 20:44:40.315523987 +0000 UTC m=+1499.994152602"
Oct 30 20:44:40.443210 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:40.443146    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf" podStartSLOduration=2.2406791249999998 podStartE2EDuration="5.443127085s" totalImagesPullingTime="3.20244796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:40.440787951 +0000 UTC m=+1500.119416584" watchObservedRunningTime="2026-10-30 20:44:40.443127085 +0000 UTC m=+1500.121755703"
Oct 30 20:44:51.209100 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:51.209070    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf"
Oct 30 20:44:51.209605 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:44:51.209237    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h"
Oct 30 20:45:00.198211 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:00.198172    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb"]
Oct 30 20:45:00.203449 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:00.203424    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb"
Oct 30 20:45:00.255009 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:00.254982    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 20:45:00.266775 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:00.266746    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a172ae51-588f-41d8-9223-aca6eac09cf0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb\" (UID: \"a172ae51-588f-41d8-9223-aca6eac09cf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb"
Oct 30 20:45:00.266927 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:00.266910    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2jbt\" (UniqueName: \"kubernetes.io/projected/a172ae51-588f-41d8-9223-aca6eac09cf0-kube-api-access-r2jbt\") pod \"maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb\" (UID: \"a172ae51-588f-41d8-9223-aca6eac09cf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb"
Oct 30 20:45:00.368235 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:00.368202    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a172ae51-588f-41d8-9223-aca6eac09cf0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb\" (UID: \"a172ae51-588f-41d8-9223-aca6eac09cf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb"
Oct 30 20:45:00.368419 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:00.368300    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2jbt\" (UniqueName: \"kubernetes.io/projected/a172ae51-588f-41d8-9223-aca6eac09cf0-kube-api-access-r2jbt\") pod \"maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb\" (UID: \"a172ae51-588f-41d8-9223-aca6eac09cf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb"
Oct 30 20:45:00.368796 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:00.368778    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a172ae51-588f-41d8-9223-aca6eac09cf0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb\" (UID: \"a172ae51-588f-41d8-9223-aca6eac09cf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb"
Oct 30 20:45:00.401857 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:00.401824    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb"]
Oct 30 20:45:00.426190 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:00.426155    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2jbt\" (UniqueName: \"kubernetes.io/projected/a172ae51-588f-41d8-9223-aca6eac09cf0-kube-api-access-r2jbt\") pod \"maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb\" (UID: \"a172ae51-588f-41d8-9223-aca6eac09cf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb"
Oct 30 20:45:00.517927 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:00.517846    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb"
Oct 30 20:45:00.893873 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:00.893841    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb"]
Oct 30 20:45:00.895248 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:45:00.895225    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda172ae51_588f_41d8_9223_aca6eac09cf0.slice/crio-b86b97d9af1b4a8b6210bf57a27292aeb7b058527bafe8d205a1d0c529214609 WatchSource:0}: Error finding container b86b97d9af1b4a8b6210bf57a27292aeb7b058527bafe8d205a1d0c529214609: Status 404 returned error can't find the container with id b86b97d9af1b4a8b6210bf57a27292aeb7b058527bafe8d205a1d0c529214609
Oct 30 20:45:00.954255 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:45:00.954226    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:45:01.374073 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:01.373977    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb" event={"ID":"a172ae51-588f-41d8-9223-aca6eac09cf0","Type":"ContainerStarted","Data":"b86b97d9af1b4a8b6210bf57a27292aeb7b058527bafe8d205a1d0c529214609"}
Oct 30 20:45:06.407140 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:06.407100    4471 generic.go:356] "Generic (PLEG): container finished" podID="a172ae51-588f-41d8-9223-aca6eac09cf0" containerID="c957baf255c5b53f5811caa33aa9d340340af56d2432eafeec18f5d2e3bada99" exitCode=60
Oct 30 20:45:06.407567 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:06.407185    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb" event={"ID":"a172ae51-588f-41d8-9223-aca6eac09cf0","Type":"ContainerDied","Data":"c957baf255c5b53f5811caa33aa9d340340af56d2432eafeec18f5d2e3bada99"}
Oct 30 20:45:06.407567 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:06.407454    4471 scope.go:122] "RemoveContainer" containerID="c957baf255c5b53f5811caa33aa9d340340af56d2432eafeec18f5d2e3bada99"
Oct 30 20:45:07.414591 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:07.414552    4471 generic.go:356] "Generic (PLEG): container finished" podID="a172ae51-588f-41d8-9223-aca6eac09cf0" containerID="6d0c234181b66b608b4a8b1c891a3c7ccc96fba19dbb9e54743b8b6eb968cab5" exitCode=60
Oct 30 20:45:07.415046 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:07.414639    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb" event={"ID":"a172ae51-588f-41d8-9223-aca6eac09cf0","Type":"ContainerDied","Data":"6d0c234181b66b608b4a8b1c891a3c7ccc96fba19dbb9e54743b8b6eb968cab5"}
Oct 30 20:45:07.415046 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:07.414752    4471 scope.go:122] "RemoveContainer" containerID="c957baf255c5b53f5811caa33aa9d340340af56d2432eafeec18f5d2e3bada99"
Oct 30 20:45:07.415046 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:07.415031    4471 scope.go:122] "RemoveContainer" containerID="6d0c234181b66b608b4a8b1c891a3c7ccc96fba19dbb9e54743b8b6eb968cab5"
Oct 30 20:45:07.415407 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:45:07.415380    4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb_odh-ai-gateway-infra(a172ae51-588f-41d8-9223-aca6eac09cf0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb" podUID="a172ae51-588f-41d8-9223-aca6eac09cf0"
Oct 30 20:45:08.027304 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:08.027266    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"]
Oct 30 20:45:08.032436 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:08.032420    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"
Oct 30 20:45:08.085306 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:08.085280    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0d757729-serving-cert\""
Oct 30 20:45:08.137574 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:08.137542    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"]
Oct 30 20:45:08.174438 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:08.174405    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvsrj\" (UniqueName: \"kubernetes.io/projected/e4cbed9b-e833-438e-b948-618ed3b525cf-kube-api-access-fvsrj\") pod \"maas-api-e2e-ait-0d757729-55c8b46766-gjktr\" (UID: \"e4cbed9b-e833-438e-b948-618ed3b525cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"
Oct 30 20:45:08.174625 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:08.174552    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4cbed9b-e833-438e-b948-618ed3b525cf-metrics-certs\") pod \"maas-api-e2e-ait-0d757729-55c8b46766-gjktr\" (UID: \"e4cbed9b-e833-438e-b948-618ed3b525cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"
Oct 30 20:45:08.174625 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:08.174591    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cbed9b-e833-438e-b948-618ed3b525cf-openshift-service-ca\") pod \"maas-api-e2e-ait-0d757729-55c8b46766-gjktr\" (UID: \"e4cbed9b-e833-438e-b948-618ed3b525cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"
Oct 30 20:45:08.174752 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:08.174715    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4cbed9b-e833-438e-b948-618ed3b525cf-maas-api-tls\") pod \"maas-api-e2e-ait-0d757729-55c8b46766-gjktr\" (UID: \"e4cbed9b-e833-438e-b948-618ed3b525cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"
Oct 30 20:45:08.298438 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:08.283385    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvsrj\" (UniqueName: \"kubernetes.io/projected/e4cbed9b-e833-438e-b948-618ed3b525cf-kube-api-access-fvsrj\") pod \"maas-api-e2e-ait-0d757729-55c8b46766-gjktr\" (UID: \"e4cbed9b-e833-438e-b948-618ed3b525cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"
Oct 30 20:45:08.316341 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:08.300380    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4cbed9b-e833-438e-b948-618ed3b525cf-metrics-certs\") pod \"maas-api-e2e-ait-0d757729-55c8b46766-gjktr\" (UID: \"e4cbed9b-e833-438e-b948-618ed3b525cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"
Oct 30 20:45:08.316848 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:08.316799    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cbed9b-e833-438e-b948-618ed3b525cf-openshift-service-ca\") pod \"maas-api-e2e-ait-0d757729-55c8b46766-gjktr\" (UID: \"e4cbed9b-e833-438e-b948-618ed3b525cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"
Oct 30 20:45:08.344510 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:08.344475    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4cbed9b-e833-438e-b948-618ed3b525cf-maas-api-tls\") pod \"maas-api-e2e-ait-0d757729-55c8b46766-gjktr\" (UID: \"e4cbed9b-e833-438e-b948-618ed3b525cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"
Oct 30 20:45:08.345479 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:08.340160    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4cbed9b-e833-438e-b948-618ed3b525cf-metrics-certs\") pod \"maas-api-e2e-ait-0d757729-55c8b46766-gjktr\" (UID: \"e4cbed9b-e833-438e-b948-618ed3b525cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"
Oct 30 20:45:08.345853 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:08.317753    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cbed9b-e833-438e-b948-618ed3b525cf-openshift-service-ca\") pod \"maas-api-e2e-ait-0d757729-55c8b46766-gjktr\" (UID: \"e4cbed9b-e833-438e-b948-618ed3b525cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"
Oct 30 20:45:08.349235 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:08.349208    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4cbed9b-e833-438e-b948-618ed3b525cf-maas-api-tls\") pod \"maas-api-e2e-ait-0d757729-55c8b46766-gjktr\" (UID: \"e4cbed9b-e833-438e-b948-618ed3b525cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"
Oct 30 20:45:08.382273 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:08.378909    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvsrj\" (UniqueName: \"kubernetes.io/projected/e4cbed9b-e833-438e-b948-618ed3b525cf-kube-api-access-fvsrj\") pod \"maas-api-e2e-ait-0d757729-55c8b46766-gjktr\" (UID: \"e4cbed9b-e833-438e-b948-618ed3b525cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"
Oct 30 20:45:08.422243 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:08.422218    4471 scope.go:122] "RemoveContainer" containerID="6d0c234181b66b608b4a8b1c891a3c7ccc96fba19dbb9e54743b8b6eb968cab5"
Oct 30 20:45:08.422632 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:45:08.422517    4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb_odh-ai-gateway-infra(a172ae51-588f-41d8-9223-aca6eac09cf0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb" podUID="a172ae51-588f-41d8-9223-aca6eac09cf0"
Oct 30 20:45:08.645768 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:08.645738    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"
Oct 30 20:45:09.020616 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:09.020588    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"]
Oct 30 20:45:09.021869 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:45:09.021838    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4cbed9b_e833_438e_b948_618ed3b525cf.slice/crio-5e960703648e628f09a761926db1fc38f86e0cb58241134c5a74da5d2be07cb4 WatchSource:0}: Error finding container 5e960703648e628f09a761926db1fc38f86e0cb58241134c5a74da5d2be07cb4: Status 404 returned error can't find the container with id 5e960703648e628f09a761926db1fc38f86e0cb58241134c5a74da5d2be07cb4
Oct 30 20:45:09.428364 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:09.428299    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr" event={"ID":"e4cbed9b-e833-438e-b948-618ed3b525cf","Type":"ContainerStarted","Data":"5e960703648e628f09a761926db1fc38f86e0cb58241134c5a74da5d2be07cb4"}
Oct 30 20:45:10.435937 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:10.435905    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr" event={"ID":"e4cbed9b-e833-438e-b948-618ed3b525cf","Type":"ContainerStarted","Data":"a465a2cc205b69a9789acbaf679bd8c5e46be65bcace8aaff8876397f4f0570b"}
Oct 30 20:45:10.436454 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:10.435972    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"
Oct 30 20:45:10.552636 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:10.552574    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr" podStartSLOduration=2.603890871 podStartE2EDuration="3.552558585s" totalImagesPullingTime="948.667714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:10.550180967 +0000 UTC m=+1530.228809585" watchObservedRunningTime="2026-10-30 20:45:10.552558585 +0000 UTC m=+1530.231187240"
Oct 30 20:45:16.450148 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:16.450116    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"
Oct 30 20:45:18.424842 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:18.424806    4471 scope.go:122] "RemoveContainer" containerID="6d0c234181b66b608b4a8b1c891a3c7ccc96fba19dbb9e54743b8b6eb968cab5"
Oct 30 20:45:19.524736 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:19.524709    4471 generic.go:356] "Generic (PLEG): container finished" podID="a172ae51-588f-41d8-9223-aca6eac09cf0" containerID="ab273d91baad59293020230b6ac0a94bd0f52309b23d3c47c96180bdb2c1d7c5" exitCode=60
Oct 30 20:45:19.525093 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:19.524768    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb" event={"ID":"a172ae51-588f-41d8-9223-aca6eac09cf0","Type":"ContainerDied","Data":"ab273d91baad59293020230b6ac0a94bd0f52309b23d3c47c96180bdb2c1d7c5"}
Oct 30 20:45:19.525093 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:19.524849    4471 scope.go:122] "RemoveContainer" containerID="6d0c234181b66b608b4a8b1c891a3c7ccc96fba19dbb9e54743b8b6eb968cab5"
Oct 30 20:45:19.525244 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:19.525229    4471 scope.go:122] "RemoveContainer" containerID="ab273d91baad59293020230b6ac0a94bd0f52309b23d3c47c96180bdb2c1d7c5"
Oct 30 20:45:19.525778 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:45:19.525585    4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb_odh-ai-gateway-infra(a172ae51-588f-41d8-9223-aca6eac09cf0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb" podUID="a172ae51-588f-41d8-9223-aca6eac09cf0"
Oct 30 20:45:20.651874 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:20.651837    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb"]
Oct 30 20:45:20.817740 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:20.817714    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb"
Oct 30 20:45:20.860943 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:20.860910    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a172ae51-588f-41d8-9223-aca6eac09cf0-kube-api-access-r2jbt\" (UniqueName: \"kubernetes.io/projected/a172ae51-588f-41d8-9223-aca6eac09cf0-kube-api-access-r2jbt\") pod \"a172ae51-588f-41d8-9223-aca6eac09cf0\" (UID: \"a172ae51-588f-41d8-9223-aca6eac09cf0\") "
Oct 30 20:45:20.861129 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:20.861054    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a172ae51-588f-41d8-9223-aca6eac09cf0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a172ae51-588f-41d8-9223-aca6eac09cf0-maas-api-service-ca\") pod \"a172ae51-588f-41d8-9223-aca6eac09cf0\" (UID: \"a172ae51-588f-41d8-9223-aca6eac09cf0\") "
Oct 30 20:45:20.861684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:20.861655    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a172ae51-588f-41d8-9223-aca6eac09cf0-maas-api-service-ca" pod "a172ae51-588f-41d8-9223-aca6eac09cf0" (UID: "a172ae51-588f-41d8-9223-aca6eac09cf0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:45:20.863385 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:20.863356    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a172ae51-588f-41d8-9223-aca6eac09cf0-kube-api-access-r2jbt" pod "a172ae51-588f-41d8-9223-aca6eac09cf0" (UID: "a172ae51-588f-41d8-9223-aca6eac09cf0"). InnerVolumeSpecName "kube-api-access-r2jbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:45:20.962995 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:20.962903    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2jbt\" (UniqueName: \"kubernetes.io/projected/a172ae51-588f-41d8-9223-aca6eac09cf0-kube-api-access-r2jbt\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:20.962995 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:20.962938    4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a172ae51-588f-41d8-9223-aca6eac09cf0-maas-api-service-ca\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:21.540387 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:21.540350    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb" event={"ID":"a172ae51-588f-41d8-9223-aca6eac09cf0","Type":"ContainerDied","Data":"b86b97d9af1b4a8b6210bf57a27292aeb7b058527bafe8d205a1d0c529214609"}
Oct 30 20:45:21.540387 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:21.540376    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb"
Oct 30 20:45:21.540589 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:21.540415    4471 scope.go:122] "RemoveContainer" containerID="ab273d91baad59293020230b6ac0a94bd0f52309b23d3c47c96180bdb2c1d7c5"
Oct 30 20:45:21.702929 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:21.702897    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb"]
Oct 30 20:45:21.752139 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:21.752109    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889885-pcjzb"]
Oct 30 20:45:22.431816 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:22.431781    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a172ae51-588f-41d8-9223-aca6eac09cf0" path="/var/lib/kubelet/pods/a172ae51-588f-41d8-9223-aca6eac09cf0/volumes"
Oct 30 20:45:30.954951 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:45:30.954910    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:45:41.523396 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:41.515050    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l"]
Oct 30 20:45:41.526153 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:41.526119    4471 cpu_manager.go:414] "containerMap: removing container" podUID="a172ae51-588f-41d8-9223-aca6eac09cf0" containerName="cleanup"
Oct 30 20:45:41.526356 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:41.526340    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a172ae51-588f-41d8-9223-aca6eac09cf0" containerName="cleanup"
Oct 30 20:45:41.526456 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:41.526447    4471 cpu_manager.go:414] "containerMap: removing container" podUID="a172ae51-588f-41d8-9223-aca6eac09cf0" containerName="cleanup"
Oct 30 20:45:41.526523 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:41.526514    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a172ae51-588f-41d8-9223-aca6eac09cf0" containerName="cleanup"
Oct 30 20:45:41.526859 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:41.526749    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="a172ae51-588f-41d8-9223-aca6eac09cf0" containerName="cleanup"
Oct 30 20:45:41.527042 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:41.527030    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="a172ae51-588f-41d8-9223-aca6eac09cf0" containerName="cleanup"
Oct 30 20:45:41.527123 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:41.527113    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="a172ae51-588f-41d8-9223-aca6eac09cf0" containerName="cleanup"
Oct 30 20:45:41.542117 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:41.537718    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l"
Oct 30 20:45:41.542117 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:41.537933    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l"]
Oct 30 20:45:41.629233 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:41.629201    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c650f2b7\""
Oct 30 20:45:41.629233 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:41.629239    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c650f2b7-dockercfg-d6jx2\""
Oct 30 20:45:41.638238 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:41.638202    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6133567-cd29-4080-a47f-72ce483242c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l\" (UID: \"b6133567-cd29-4080-a47f-72ce483242c9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l"
Oct 30 20:45:41.638436 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:41.638251    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9fx7\" (UniqueName: \"kubernetes.io/projected/b6133567-cd29-4080-a47f-72ce483242c9-kube-api-access-j9fx7\") pod \"payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l\" (UID: \"b6133567-cd29-4080-a47f-72ce483242c9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l"
Oct 30 20:45:41.740703 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:41.740655    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6133567-cd29-4080-a47f-72ce483242c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l\" (UID: \"b6133567-cd29-4080-a47f-72ce483242c9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l"
Oct 30 20:45:41.740866 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:41.740713    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9fx7\" (UniqueName: \"kubernetes.io/projected/b6133567-cd29-4080-a47f-72ce483242c9-kube-api-access-j9fx7\") pod \"payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l\" (UID: \"b6133567-cd29-4080-a47f-72ce483242c9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l"
Oct 30 20:45:41.741455 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:41.741435    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6133567-cd29-4080-a47f-72ce483242c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l\" (UID: \"b6133567-cd29-4080-a47f-72ce483242c9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l"
Oct 30 20:45:41.802292 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:41.802219    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9fx7\" (UniqueName: \"kubernetes.io/projected/b6133567-cd29-4080-a47f-72ce483242c9-kube-api-access-j9fx7\") pod \"payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l\" (UID: \"b6133567-cd29-4080-a47f-72ce483242c9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l"
Oct 30 20:45:41.861653 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:41.861619    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l"
Oct 30 20:45:42.243822 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:42.243793    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l"]
Oct 30 20:45:42.246155 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:45:42.246126    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6133567_cd29_4080_a47f_72ce483242c9.slice/crio-1d9a0c90ffa1a07c7c3d2670b004759f69a3b16e7722756139e4151187397287 WatchSource:0}: Error finding container 1d9a0c90ffa1a07c7c3d2670b004759f69a3b16e7722756139e4151187397287: Status 404 returned error can't find the container with id 1d9a0c90ffa1a07c7c3d2670b004759f69a3b16e7722756139e4151187397287
Oct 30 20:45:43.042152 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:43.042113    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l" event={"ID":"b6133567-cd29-4080-a47f-72ce483242c9","Type":"ContainerStarted","Data":"ead845c7571ef9b1e38253d859f5782492dc139d2233d8079a15f2da88499410"}
Oct 30 20:45:43.042644 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:43.042222    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l" event={"ID":"b6133567-cd29-4080-a47f-72ce483242c9","Type":"ContainerStarted","Data":"1d9a0c90ffa1a07c7c3d2670b004759f69a3b16e7722756139e4151187397287"}
Oct 30 20:45:43.042644 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:43.042257    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l"
Oct 30 20:45:43.162124 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:43.162074    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l" podStartSLOduration=2.162060281 podStartE2EDuration="2.162060281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:43.158959446 +0000 UTC m=+1562.837588062" watchObservedRunningTime="2026-10-30 20:45:43.162060281 +0000 UTC m=+1562.840688917"
Oct 30 20:45:46.434063 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:46.432108    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng"]
Oct 30 20:45:46.435436 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:46.435408    4471 cpu_manager.go:414] "containerMap: removing container" podUID="a172ae51-588f-41d8-9223-aca6eac09cf0" containerName="cleanup"
Oct 30 20:45:46.435595 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:46.435583    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a172ae51-588f-41d8-9223-aca6eac09cf0" containerName="cleanup"
Oct 30 20:45:46.476344 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:46.473770    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng"
Oct 30 20:45:46.497527 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:46.497493    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng"]
Oct 30 20:45:46.538298 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:46.533920    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d4bdac-79bc-48af-b3ac-4141651c0f86-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng\" (UID: \"38d4bdac-79bc-48af-b3ac-4141651c0f86\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng"
Oct 30 20:45:46.539029 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:46.538999    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqx8k\" (UniqueName: \"kubernetes.io/projected/38d4bdac-79bc-48af-b3ac-4141651c0f86-kube-api-access-qqx8k\") pod \"payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng\" (UID: \"38d4bdac-79bc-48af-b3ac-4141651c0f86\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng"
Oct 30 20:45:46.584468 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:46.580849    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-d71c23\""
Oct 30 20:45:46.584468 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:46.581009    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-d71c23-dockercfg-qgpz5\""
Oct 30 20:45:46.640483 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:46.640448    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d4bdac-79bc-48af-b3ac-4141651c0f86-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng\" (UID: \"38d4bdac-79bc-48af-b3ac-4141651c0f86\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng"
Oct 30 20:45:46.640680 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:46.640653    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqx8k\" (UniqueName: \"kubernetes.io/projected/38d4bdac-79bc-48af-b3ac-4141651c0f86-kube-api-access-qqx8k\") pod \"payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng\" (UID: \"38d4bdac-79bc-48af-b3ac-4141651c0f86\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng"
Oct 30 20:45:46.641289 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:46.641263    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d4bdac-79bc-48af-b3ac-4141651c0f86-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng\" (UID: \"38d4bdac-79bc-48af-b3ac-4141651c0f86\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng"
Oct 30 20:45:46.697773 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:46.697699    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqx8k\" (UniqueName: \"kubernetes.io/projected/38d4bdac-79bc-48af-b3ac-4141651c0f86-kube-api-access-qqx8k\") pod \"payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng\" (UID: \"38d4bdac-79bc-48af-b3ac-4141651c0f86\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng"
Oct 30 20:45:46.886186 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:46.886154    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng"
Oct 30 20:45:47.268243 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:47.268220    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng"]
Oct 30 20:45:47.269970 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:45:47.269945    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d4bdac_79bc_48af_b3ac_4141651c0f86.slice/crio-bc6bdc0eb04d156112bcbb8496276e0f4cbbcca3818c5a385228d9894f3e65be WatchSource:0}: Error finding container bc6bdc0eb04d156112bcbb8496276e0f4cbbcca3818c5a385228d9894f3e65be: Status 404 returned error can't find the container with id bc6bdc0eb04d156112bcbb8496276e0f4cbbcca3818c5a385228d9894f3e65be
Oct 30 20:45:48.118910 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:48.118869    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng" event={"ID":"38d4bdac-79bc-48af-b3ac-4141651c0f86","Type":"ContainerStarted","Data":"7d826776e5c0276cc5caa472c1bb7efce0c7b015c135d9a5f617cf4ff33a26e9"}
Oct 30 20:45:48.119550 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:48.119522    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng" event={"ID":"38d4bdac-79bc-48af-b3ac-4141651c0f86","Type":"ContainerStarted","Data":"bc6bdc0eb04d156112bcbb8496276e0f4cbbcca3818c5a385228d9894f3e65be"}
Oct 30 20:45:48.120828 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:48.120807    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng"
Oct 30 20:45:48.236529 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:48.236478    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng" podStartSLOduration=2.236462919 podStartE2EDuration="2.236462919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:48.23496163 +0000 UTC m=+1567.913590246" watchObservedRunningTime="2026-10-30 20:45:48.236462919 +0000 UTC m=+1567.915091535"
Oct 30 20:45:54.050868 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:45:54.050777    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l"
Oct 30 20:46:00.136089 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:00.136059    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng"
Oct 30 20:46:00.955923 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:46:00.955890    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:46:23.468530 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:23.468474    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf"]
Oct 30 20:46:23.582517 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:23.582485    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h"]
Oct 30 20:46:27.507694 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:27.507661    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"]
Oct 30 20:46:27.508195 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:27.507875    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m" podUID="bba3ab4d-cb5f-45dc-9269-31145e9a396d" containerName="istio-proxy" containerID="cri-o://ff10320d35b9216cfae7f772b91ff37a2db4287679e603303f0cbe0b443eed90" gracePeriod=30
Oct 30 20:46:28.402373 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:28.402316    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"]
Oct 30 20:46:28.408583 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:28.408554    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"
Oct 30 20:46:28.456355 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:28.456297    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbktw\" (UniqueName: \"kubernetes.io/projected/fce24575-c3b1-4a12-a5be-43ead65afa74-kube-api-access-vbktw\") pod \"maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt\" (UID: \"fce24575-c3b1-4a12-a5be-43ead65afa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"
Oct 30 20:46:28.456506 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:28.456464    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fce24575-c3b1-4a12-a5be-43ead65afa74-metrics-certs\") pod \"maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt\" (UID: \"fce24575-c3b1-4a12-a5be-43ead65afa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"
Oct 30 20:46:28.456551 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:28.456529    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fce24575-c3b1-4a12-a5be-43ead65afa74-maas-api-tls\") pod \"maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt\" (UID: \"fce24575-c3b1-4a12-a5be-43ead65afa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"
Oct 30 20:46:28.456610 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:28.456593    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce24575-c3b1-4a12-a5be-43ead65afa74-openshift-service-ca\") pod \"maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt\" (UID: \"fce24575-c3b1-4a12-a5be-43ead65afa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"
Oct 30 20:46:28.472622 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:28.472567    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf" podUID="e6178eb2-618b-4e28-bff5-86952e4965a7" containerName="payload-processing" containerID="cri-o://6c3e4313f643d6801f6de39f2a87f57c24b98e2967117608fed6901d643db1ec" gracePeriod=25
Oct 30 20:46:28.509887 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:28.509862    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c650f2b7-serving-cert\""
Oct 30 20:46:28.515066 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:28.515039    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"]
Oct 30 20:46:28.557806 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:28.557769    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fce24575-c3b1-4a12-a5be-43ead65afa74-metrics-certs\") pod \"maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt\" (UID: \"fce24575-c3b1-4a12-a5be-43ead65afa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"
Oct 30 20:46:28.557973 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:28.557843    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fce24575-c3b1-4a12-a5be-43ead65afa74-maas-api-tls\") pod \"maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt\" (UID: \"fce24575-c3b1-4a12-a5be-43ead65afa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"
Oct 30 20:46:28.557973 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:28.557876    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce24575-c3b1-4a12-a5be-43ead65afa74-openshift-service-ca\") pod \"maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt\" (UID: \"fce24575-c3b1-4a12-a5be-43ead65afa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"
Oct 30 20:46:28.558114 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:28.558096    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbktw\" (UniqueName: \"kubernetes.io/projected/fce24575-c3b1-4a12-a5be-43ead65afa74-kube-api-access-vbktw\") pod \"maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt\" (UID: \"fce24575-c3b1-4a12-a5be-43ead65afa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"
Oct 30 20:46:28.559057 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:28.559025    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce24575-c3b1-4a12-a5be-43ead65afa74-openshift-service-ca\") pod \"maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt\" (UID: \"fce24575-c3b1-4a12-a5be-43ead65afa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"
Oct 30 20:46:28.559200 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:46:28.559186    4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-serving-cert: secret "maas-api-e2e-mt-c650f2b7-serving-cert" not found
Oct 30 20:46:28.559263 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:46:28.559253    4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fce24575-c3b1-4a12-a5be-43ead65afa74-maas-api-tls podName:fce24575-c3b1-4a12-a5be-43ead65afa74 nodeName:}" failed. No retries permitted until 2026-10-30 20:46:29.059235344 +0000 UTC m=+1608.737863943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fce24575-c3b1-4a12-a5be-43ead65afa74-maas-api-tls") pod "maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt" (UID: "fce24575-c3b1-4a12-a5be-43ead65afa74") : secret "maas-api-e2e-mt-c650f2b7-serving-cert" not found
Oct 30 20:46:28.560794 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:28.560769    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fce24575-c3b1-4a12-a5be-43ead65afa74-metrics-certs\") pod \"maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt\" (UID: \"fce24575-c3b1-4a12-a5be-43ead65afa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"
Oct 30 20:46:28.583694 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:28.583661    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h" podUID="3e1cfa20-1a50-46be-9967-cd4c6f8075bc" containerName="payload-pre-processing" containerID="cri-o://8a59b0fb4ab3a948c6ba83bc2aaac7b6fe80cb56c32980259e1122d9d2f18954" gracePeriod=25
Oct 30 20:46:28.615402 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:28.615370    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbktw\" (UniqueName: \"kubernetes.io/projected/fce24575-c3b1-4a12-a5be-43ead65afa74-kube-api-access-vbktw\") pod \"maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt\" (UID: \"fce24575-c3b1-4a12-a5be-43ead65afa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"
Oct 30 20:46:28.940553 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:28.940517    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf"
Oct 30 20:46:28.991578 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:28.991543    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6178eb2-618b-4e28-bff5-86952e4965a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6178eb2-618b-4e28-bff5-86952e4965a7-plugins-config-volume\") pod \"e6178eb2-618b-4e28-bff5-86952e4965a7\" (UID: \"e6178eb2-618b-4e28-bff5-86952e4965a7\") "
Oct 30 20:46:28.991835 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:28.991598    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6178eb2-618b-4e28-bff5-86952e4965a7-kube-api-access-lrjcm\" (UniqueName: \"kubernetes.io/projected/e6178eb2-618b-4e28-bff5-86952e4965a7-kube-api-access-lrjcm\") pod \"e6178eb2-618b-4e28-bff5-86952e4965a7\" (UID: \"e6178eb2-618b-4e28-bff5-86952e4965a7\") "
Oct 30 20:46:28.992355 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:28.992296    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6178eb2-618b-4e28-bff5-86952e4965a7-plugins-config-volume" pod "e6178eb2-618b-4e28-bff5-86952e4965a7" (UID: "e6178eb2-618b-4e28-bff5-86952e4965a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:46:28.995414 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:28.995392    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6178eb2-618b-4e28-bff5-86952e4965a7-kube-api-access-lrjcm" pod "e6178eb2-618b-4e28-bff5-86952e4965a7" (UID: "e6178eb2-618b-4e28-bff5-86952e4965a7"). InnerVolumeSpecName "kube-api-access-lrjcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:46:29.040238 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.040215    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h"
Oct 30 20:46:29.093466 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.093423    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e1cfa20-1a50-46be-9967-cd4c6f8075bc-kube-api-access-qsd88\" (UniqueName: \"kubernetes.io/projected/3e1cfa20-1a50-46be-9967-cd4c6f8075bc-kube-api-access-qsd88\") pod \"3e1cfa20-1a50-46be-9967-cd4c6f8075bc\" (UID: \"3e1cfa20-1a50-46be-9967-cd4c6f8075bc\") "
Oct 30 20:46:29.093648 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.093491    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e1cfa20-1a50-46be-9967-cd4c6f8075bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e1cfa20-1a50-46be-9967-cd4c6f8075bc-plugins-config-volume\") pod \"3e1cfa20-1a50-46be-9967-cd4c6f8075bc\" (UID: \"3e1cfa20-1a50-46be-9967-cd4c6f8075bc\") "
Oct 30 20:46:29.093886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.093862    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e1cfa20-1a50-46be-9967-cd4c6f8075bc-plugins-config-volume" pod "3e1cfa20-1a50-46be-9967-cd4c6f8075bc" (UID: "3e1cfa20-1a50-46be-9967-cd4c6f8075bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:46:29.094040 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.094023    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fce24575-c3b1-4a12-a5be-43ead65afa74-maas-api-tls\") pod \"maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt\" (UID: \"fce24575-c3b1-4a12-a5be-43ead65afa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"
Oct 30 20:46:29.094181 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.094167    4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e1cfa20-1a50-46be-9967-cd4c6f8075bc-plugins-config-volume\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:29.094256 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.094183    4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6178eb2-618b-4e28-bff5-86952e4965a7-plugins-config-volume\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:29.094256 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.094191    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrjcm\" (UniqueName: \"kubernetes.io/projected/e6178eb2-618b-4e28-bff5-86952e4965a7-kube-api-access-lrjcm\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:29.096364 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.096304    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e1cfa20-1a50-46be-9967-cd4c6f8075bc-kube-api-access-qsd88" pod "3e1cfa20-1a50-46be-9967-cd4c6f8075bc" (UID: "3e1cfa20-1a50-46be-9967-cd4c6f8075bc"). InnerVolumeSpecName "kube-api-access-qsd88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:46:29.096740 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.096722    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fce24575-c3b1-4a12-a5be-43ead65afa74-maas-api-tls\") pod \"maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt\" (UID: \"fce24575-c3b1-4a12-a5be-43ead65afa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"
Oct 30 20:46:29.195037 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.195003    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsd88\" (UniqueName: \"kubernetes.io/projected/3e1cfa20-1a50-46be-9967-cd4c6f8075bc-kube-api-access-qsd88\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:29.323150 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.323079    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"
Oct 30 20:46:29.398146 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.398114    4471 generic.go:356] "Generic (PLEG): container finished" podID="e6178eb2-618b-4e28-bff5-86952e4965a7" containerID="6c3e4313f643d6801f6de39f2a87f57c24b98e2967117608fed6901d643db1ec" exitCode=0
Oct 30 20:46:29.398335 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.398167    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf"
Oct 30 20:46:29.398335 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.398191    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf" event={"ID":"e6178eb2-618b-4e28-bff5-86952e4965a7","Type":"ContainerDied","Data":"6c3e4313f643d6801f6de39f2a87f57c24b98e2967117608fed6901d643db1ec"}
Oct 30 20:46:29.398335 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.398250    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf" event={"ID":"e6178eb2-618b-4e28-bff5-86952e4965a7","Type":"ContainerDied","Data":"d4ee0ada23935a5c71c5bbe3313c8317b78438e1e6f86f60943019459032dc0d"}
Oct 30 20:46:29.398335 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.398280    4471 scope.go:122] "RemoveContainer" containerID="6c3e4313f643d6801f6de39f2a87f57c24b98e2967117608fed6901d643db1ec"
Oct 30 20:46:29.400276 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.400252    4471 generic.go:356] "Generic (PLEG): container finished" podID="3e1cfa20-1a50-46be-9967-cd4c6f8075bc" containerID="8a59b0fb4ab3a948c6ba83bc2aaac7b6fe80cb56c32980259e1122d9d2f18954" exitCode=0
Oct 30 20:46:29.400534 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.400512    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h" event={"ID":"3e1cfa20-1a50-46be-9967-cd4c6f8075bc","Type":"ContainerDied","Data":"8a59b0fb4ab3a948c6ba83bc2aaac7b6fe80cb56c32980259e1122d9d2f18954"}
Oct 30 20:46:29.400616 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.400565    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h" event={"ID":"3e1cfa20-1a50-46be-9967-cd4c6f8075bc","Type":"ContainerDied","Data":"dba08dd4dd62946df144e7b975da4a6ab6f2aa8a82c043aa49016f8db4673d91"}
Oct 30 20:46:29.400678 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.400639    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h"
Oct 30 20:46:29.411420 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.411403    4471 scope.go:122] "RemoveContainer" containerID="6c3e4313f643d6801f6de39f2a87f57c24b98e2967117608fed6901d643db1ec"
Oct 30 20:46:29.411728 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:46:29.411700    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c3e4313f643d6801f6de39f2a87f57c24b98e2967117608fed6901d643db1ec\": container with ID starting with 6c3e4313f643d6801f6de39f2a87f57c24b98e2967117608fed6901d643db1ec not found: ID does not exist" containerID="6c3e4313f643d6801f6de39f2a87f57c24b98e2967117608fed6901d643db1ec"
Oct 30 20:46:29.411799 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.411736    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3e4313f643d6801f6de39f2a87f57c24b98e2967117608fed6901d643db1ec"} err="failed to get container status \"6c3e4313f643d6801f6de39f2a87f57c24b98e2967117608fed6901d643db1ec\": rpc error: code = NotFound desc = could not find container \"6c3e4313f643d6801f6de39f2a87f57c24b98e2967117608fed6901d643db1ec\": container with ID starting with 6c3e4313f643d6801f6de39f2a87f57c24b98e2967117608fed6901d643db1ec not found: ID does not exist"
Oct 30 20:46:29.411799 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.411753    4471 scope.go:122] "RemoveContainer" containerID="8a59b0fb4ab3a948c6ba83bc2aaac7b6fe80cb56c32980259e1122d9d2f18954"
Oct 30 20:46:29.526474 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.526444    4471 scope.go:122] "RemoveContainer" containerID="8a59b0fb4ab3a948c6ba83bc2aaac7b6fe80cb56c32980259e1122d9d2f18954"
Oct 30 20:46:29.528892 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:46:29.528857    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a59b0fb4ab3a948c6ba83bc2aaac7b6fe80cb56c32980259e1122d9d2f18954\": container with ID starting with 8a59b0fb4ab3a948c6ba83bc2aaac7b6fe80cb56c32980259e1122d9d2f18954 not found: ID does not exist" containerID="8a59b0fb4ab3a948c6ba83bc2aaac7b6fe80cb56c32980259e1122d9d2f18954"
Oct 30 20:46:29.529038 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.528903    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a59b0fb4ab3a948c6ba83bc2aaac7b6fe80cb56c32980259e1122d9d2f18954"} err="failed to get container status \"8a59b0fb4ab3a948c6ba83bc2aaac7b6fe80cb56c32980259e1122d9d2f18954\": rpc error: code = NotFound desc = could not find container \"8a59b0fb4ab3a948c6ba83bc2aaac7b6fe80cb56c32980259e1122d9d2f18954\": container with ID starting with 8a59b0fb4ab3a948c6ba83bc2aaac7b6fe80cb56c32980259e1122d9d2f18954 not found: ID does not exist"
Oct 30 20:46:29.653131 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.653098    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h"]
Oct 30 20:46:29.664843 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.664817    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c2bdb58a-699df9cc79-hlk2h"]
Oct 30 20:46:29.779154 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.779125    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"]
Oct 30 20:46:29.780404 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:46:29.780381    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce24575_c3b1_4a12_a5be_43ead65afa74.slice/crio-a1d1ce87643e922dae330af654d5c7cf3e19db58039a0df9aa8f0894cd8e401f WatchSource:0}: Error finding container a1d1ce87643e922dae330af654d5c7cf3e19db58039a0df9aa8f0894cd8e401f: Status 404 returned error can't find the container with id a1d1ce87643e922dae330af654d5c7cf3e19db58039a0df9aa8f0894cd8e401f
Oct 30 20:46:29.781786 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.781770    4471 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:46:29.813914 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.813889    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf"]
Oct 30 20:46:29.821314 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:29.821292    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c2bdb58a-65b4844974-vkrzf"]
Oct 30 20:46:30.409427 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:30.409395    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt" event={"ID":"fce24575-c3b1-4a12-a5be-43ead65afa74","Type":"ContainerStarted","Data":"a1d1ce87643e922dae330af654d5c7cf3e19db58039a0df9aa8f0894cd8e401f"}
Oct 30 20:46:30.432397 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:30.432362    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e1cfa20-1a50-46be-9967-cd4c6f8075bc" path="/var/lib/kubelet/pods/3e1cfa20-1a50-46be-9967-cd4c6f8075bc/volumes"
Oct 30 20:46:30.432904 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:30.432887    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6178eb2-618b-4e28-bff5-86952e4965a7" path="/var/lib/kubelet/pods/e6178eb2-618b-4e28-bff5-86952e4965a7/volumes"
Oct 30 20:46:30.955180 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:46:30.955093    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:46:31.416832 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:31.416795    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt" event={"ID":"fce24575-c3b1-4a12-a5be-43ead65afa74","Type":"ContainerStarted","Data":"0d01a065737520cf0b1d7bba6592cc51bf9d4f537a2eaa7b430658317f163c67"}
Oct 30 20:46:31.417022 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:31.416895    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"
Oct 30 20:46:31.540345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:31.537700    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt" podStartSLOduration=2.635172446 podStartE2EDuration="3.537682046s" totalImagesPullingTime="902.5096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:31.533771136 +0000 UTC m=+1611.212399756" watchObservedRunningTime="2026-10-30 20:46:31.537682046 +0000 UTC m=+1611.216310663"
Oct 30 20:46:32.783315 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.783257    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:46:32.861441 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.861407    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bba3ab4d-cb5f-45dc-9269-31145e9a396d-kube-api-access-n2lzr\" (UniqueName: \"kubernetes.io/projected/bba3ab4d-cb5f-45dc-9269-31145e9a396d-kube-api-access-n2lzr\") pod \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") "
Oct 30 20:46:32.861631 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.861471    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-podinfo\") pod \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") "
Oct 30 20:46:32.861631 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.861508    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-data\") pod \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") "
Oct 30 20:46:32.861631 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.861584    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-token\" (UniqueName: \"kubernetes.io/projected/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-token\") pod \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") "
Oct 30 20:46:32.861631 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.861604    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-workload-certs\") pod \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") "
Oct 30 20:46:32.861859 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.861695    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-envoy\") pod \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") "
Oct 30 20:46:32.861859 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.861738    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-workload-socket\") pod \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") "
Oct 30 20:46:32.861859 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.861783    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istiod-ca-cert\") pod \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") "
Oct 30 20:46:32.861859 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.861837    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-credential-socket\") pod \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\" (UID: \"bba3ab4d-cb5f-45dc-9269-31145e9a396d\") "
Oct 30 20:46:32.862067 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.861906    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-data" pod "bba3ab4d-cb5f-45dc-9269-31145e9a396d" (UID: "bba3ab4d-cb5f-45dc-9269-31145e9a396d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:46:32.862067 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.861949    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-workload-certs" pod "bba3ab4d-cb5f-45dc-9269-31145e9a396d" (UID: "bba3ab4d-cb5f-45dc-9269-31145e9a396d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:46:32.862170 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.862092    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-workload-socket" pod "bba3ab4d-cb5f-45dc-9269-31145e9a396d" (UID: "bba3ab4d-cb5f-45dc-9269-31145e9a396d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:46:32.862407 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.862270    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-credential-socket" pod "bba3ab4d-cb5f-45dc-9269-31145e9a396d" (UID: "bba3ab4d-cb5f-45dc-9269-31145e9a396d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:46:32.862475 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.862457    4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-data\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:32.862475 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.862459    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istiod-ca-cert" pod "bba3ab4d-cb5f-45dc-9269-31145e9a396d" (UID: "bba3ab4d-cb5f-45dc-9269-31145e9a396d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:46:32.862592 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.862479    4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-workload-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:32.862592 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.862494    4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-workload-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:32.862592 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.862510    4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-credential-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:32.863864 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.863837    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-podinfo" pod "bba3ab4d-cb5f-45dc-9269-31145e9a396d" (UID: "bba3ab4d-cb5f-45dc-9269-31145e9a396d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:46:32.863979 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.863903    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba3ab4d-cb5f-45dc-9269-31145e9a396d-kube-api-access-n2lzr" pod "bba3ab4d-cb5f-45dc-9269-31145e9a396d" (UID: "bba3ab4d-cb5f-45dc-9269-31145e9a396d"). InnerVolumeSpecName "kube-api-access-n2lzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:46:32.864304 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.864285    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-token" pod "bba3ab4d-cb5f-45dc-9269-31145e9a396d" (UID: "bba3ab4d-cb5f-45dc-9269-31145e9a396d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:46:32.864366 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.864289    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-envoy" pod "bba3ab4d-cb5f-45dc-9269-31145e9a396d" (UID: "bba3ab4d-cb5f-45dc-9269-31145e9a396d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:46:32.964093 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.964056    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2lzr\" (UniqueName: \"kubernetes.io/projected/bba3ab4d-cb5f-45dc-9269-31145e9a396d-kube-api-access-n2lzr\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:32.964093 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.964095    4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-podinfo\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:32.964356 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.964111    4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-token\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:32.964356 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.964127    4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istio-envoy\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:32.964356 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:32.964139    4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bba3ab4d-cb5f-45dc-9269-31145e9a396d-istiod-ca-cert\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:33.431084 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:33.431047    4471 generic.go:356] "Generic (PLEG): container finished" podID="bba3ab4d-cb5f-45dc-9269-31145e9a396d" containerID="ff10320d35b9216cfae7f772b91ff37a2db4287679e603303f0cbe0b443eed90" exitCode=0
Oct 30 20:46:33.431289 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:33.431095    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"
Oct 30 20:46:33.431289 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:33.431093    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m" event={"ID":"bba3ab4d-cb5f-45dc-9269-31145e9a396d","Type":"ContainerDied","Data":"ff10320d35b9216cfae7f772b91ff37a2db4287679e603303f0cbe0b443eed90"}
Oct 30 20:46:33.431289 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:33.431234    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m" event={"ID":"bba3ab4d-cb5f-45dc-9269-31145e9a396d","Type":"ContainerDied","Data":"729e9fe28cb14df31b75a74e602439bb494e85c5054bab7b6333dec403714294"}
Oct 30 20:46:33.431289 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:33.431270    4471 scope.go:122] "RemoveContainer" containerID="ff10320d35b9216cfae7f772b91ff37a2db4287679e603303f0cbe0b443eed90"
Oct 30 20:46:33.442232 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:33.442215    4471 scope.go:122] "RemoveContainer" containerID="ff10320d35b9216cfae7f772b91ff37a2db4287679e603303f0cbe0b443eed90"
Oct 30 20:46:33.442481 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:46:33.442457    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff10320d35b9216cfae7f772b91ff37a2db4287679e603303f0cbe0b443eed90\": container with ID starting with ff10320d35b9216cfae7f772b91ff37a2db4287679e603303f0cbe0b443eed90 not found: ID does not exist" containerID="ff10320d35b9216cfae7f772b91ff37a2db4287679e603303f0cbe0b443eed90"
Oct 30 20:46:33.442544 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:33.442489    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff10320d35b9216cfae7f772b91ff37a2db4287679e603303f0cbe0b443eed90"} err="failed to get container status \"ff10320d35b9216cfae7f772b91ff37a2db4287679e603303f0cbe0b443eed90\": rpc error: code = NotFound desc = could not find container \"ff10320d35b9216cfae7f772b91ff37a2db4287679e603303f0cbe0b443eed90\": container with ID starting with ff10320d35b9216cfae7f772b91ff37a2db4287679e603303f0cbe0b443eed90 not found: ID does not exist"
Oct 30 20:46:33.594216 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:33.594184    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"]
Oct 30 20:46:33.661375 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:33.654626    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a78bf4-openshift-default-79b556ccdc-lgn2m"]
Oct 30 20:46:34.431992 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:34.431947    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bba3ab4d-cb5f-45dc-9269-31145e9a396d" path="/var/lib/kubelet/pods/bba3ab4d-cb5f-45dc-9269-31145e9a396d/volumes"
Oct 30 20:46:37.427405 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:37.427375    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"
Oct 30 20:46:38.012058 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.012022    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"]
Oct 30 20:46:38.012586 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.012571    4471 cpu_manager.go:414] "containerMap: removing container" podUID="bba3ab4d-cb5f-45dc-9269-31145e9a396d" containerName="istio-proxy"
Oct 30 20:46:38.012641 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.012589    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bba3ab4d-cb5f-45dc-9269-31145e9a396d" containerName="istio-proxy"
Oct 30 20:46:38.012641 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.012598    4471 cpu_manager.go:414] "containerMap: removing container" podUID="e6178eb2-618b-4e28-bff5-86952e4965a7" containerName="payload-processing"
Oct 30 20:46:38.012641 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.012603    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6178eb2-618b-4e28-bff5-86952e4965a7" containerName="payload-processing"
Oct 30 20:46:38.012641 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.012629    4471 cpu_manager.go:414] "containerMap: removing container" podUID="3e1cfa20-1a50-46be-9967-cd4c6f8075bc" containerName="payload-pre-processing"
Oct 30 20:46:38.012641 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.012634    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e1cfa20-1a50-46be-9967-cd4c6f8075bc" containerName="payload-pre-processing"
Oct 30 20:46:38.012793 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.012718    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e1cfa20-1a50-46be-9967-cd4c6f8075bc" containerName="payload-pre-processing"
Oct 30 20:46:38.012793 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.012729    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="bba3ab4d-cb5f-45dc-9269-31145e9a396d" containerName="istio-proxy"
Oct 30 20:46:38.012793 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.012733    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6178eb2-618b-4e28-bff5-86952e4965a7" containerName="payload-processing"
Oct 30 20:46:38.017535 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.017520    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"
Oct 30 20:46:38.071270 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.071240    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-d71c23-serving-cert\""
Oct 30 20:46:38.076144 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.076120    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"]
Oct 30 20:46:38.132692 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.132659    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp\" (UID: \"3c6e1eb7-98b5-436a-b84f-a2f4528904dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"
Oct 30 20:46:38.132868 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.132799    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp\" (UID: \"3c6e1eb7-98b5-436a-b84f-a2f4528904dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"
Oct 30 20:46:38.132947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.132894    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp\" (UID: \"3c6e1eb7-98b5-436a-b84f-a2f4528904dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"
Oct 30 20:46:38.132947 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.132936    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnz64\" (UniqueName: \"kubernetes.io/projected/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-kube-api-access-cnz64\") pod \"maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp\" (UID: \"3c6e1eb7-98b5-436a-b84f-a2f4528904dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"
Oct 30 20:46:38.234541 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.234506    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp\" (UID: \"3c6e1eb7-98b5-436a-b84f-a2f4528904dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"
Oct 30 20:46:38.234541 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.234544    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnz64\" (UniqueName: \"kubernetes.io/projected/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-kube-api-access-cnz64\") pod \"maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp\" (UID: \"3c6e1eb7-98b5-436a-b84f-a2f4528904dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"
Oct 30 20:46:38.234761 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.234708    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp\" (UID: \"3c6e1eb7-98b5-436a-b84f-a2f4528904dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"
Oct 30 20:46:38.234827 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.234760    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp\" (UID: \"3c6e1eb7-98b5-436a-b84f-a2f4528904dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"
Oct 30 20:46:38.234901 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:46:38.234883    4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-serving-cert: secret "maas-api-e2e-shared-a-w1-d71c23-serving-cert" not found
Oct 30 20:46:38.234973 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:46:38.234962    4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-maas-api-tls podName:3c6e1eb7-98b5-436a-b84f-a2f4528904dc nodeName:}" failed. No retries permitted until 2026-10-30 20:46:38.734944108 +0000 UTC m=+1618.413572702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-maas-api-tls") pod "maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp" (UID: "3c6e1eb7-98b5-436a-b84f-a2f4528904dc") : secret "maas-api-e2e-shared-a-w1-d71c23-serving-cert" not found
Oct 30 20:46:38.235206 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.235185    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp\" (UID: \"3c6e1eb7-98b5-436a-b84f-a2f4528904dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"
Oct 30 20:46:38.237376 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.237352    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp\" (UID: \"3c6e1eb7-98b5-436a-b84f-a2f4528904dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"
Oct 30 20:46:38.315562 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.315484    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnz64\" (UniqueName: \"kubernetes.io/projected/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-kube-api-access-cnz64\") pod \"maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp\" (UID: \"3c6e1eb7-98b5-436a-b84f-a2f4528904dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"
Oct 30 20:46:38.754335 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.754295    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp\" (UID: \"3c6e1eb7-98b5-436a-b84f-a2f4528904dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"
Oct 30 20:46:38.756951 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.756923    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp\" (UID: \"3c6e1eb7-98b5-436a-b84f-a2f4528904dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"
Oct 30 20:46:38.930374 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:38.930341    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"
Oct 30 20:46:39.311398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:39.311373    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"]
Oct 30 20:46:39.311995 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:46:39.311968    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6e1eb7_98b5_436a_b84f_a2f4528904dc.slice/crio-e585fe4c65e0629236a6648e59f31245cf66f0e5eb38798781184f3bb33ce293 WatchSource:0}: Error finding container e585fe4c65e0629236a6648e59f31245cf66f0e5eb38798781184f3bb33ce293: Status 404 returned error can't find the container with id e585fe4c65e0629236a6648e59f31245cf66f0e5eb38798781184f3bb33ce293
Oct 30 20:46:39.466370 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:39.466331    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp" event={"ID":"3c6e1eb7-98b5-436a-b84f-a2f4528904dc","Type":"ContainerStarted","Data":"e585fe4c65e0629236a6648e59f31245cf66f0e5eb38798781184f3bb33ce293"}
Oct 30 20:46:40.473847 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:40.473760    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp" event={"ID":"3c6e1eb7-98b5-436a-b84f-a2f4528904dc","Type":"ContainerStarted","Data":"4045f3371a60a44990d0c603a6e0d5e39e6af1e83670e2240799a7c4b30c317e"}
Oct 30 20:46:40.474244 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:40.473885    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"
Oct 30 20:46:40.590274 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:40.590221    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp" podStartSLOduration=2.717529357 podStartE2EDuration="3.590206417s" totalImagesPullingTime="872.67706ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:40.588155099 +0000 UTC m=+1620.266783715" watchObservedRunningTime="2026-10-30 20:46:40.590206417 +0000 UTC m=+1620.268835032"
Oct 30 20:46:46.485412 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:46.485383    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"
Oct 30 20:46:56.768074 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:56.768043    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"]
Oct 30 20:46:56.768601 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:56.768261    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr" podUID="e4cbed9b-e833-438e-b948-618ed3b525cf" containerName="maas-api" containerID="cri-o://a465a2cc205b69a9789acbaf679bd8c5e46be65bcace8aaff8876397f4f0570b" gracePeriod=30
Oct 30 20:46:57.165932 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:57.165902    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"
Oct 30 20:46:57.273914 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:57.273866    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4cbed9b-e833-438e-b948-618ed3b525cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4cbed9b-e833-438e-b948-618ed3b525cf-maas-api-tls\") pod \"e4cbed9b-e833-438e-b948-618ed3b525cf\" (UID: \"e4cbed9b-e833-438e-b948-618ed3b525cf\") "
Oct 30 20:46:57.274426 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:57.274384    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4cbed9b-e833-438e-b948-618ed3b525cf-kube-api-access-fvsrj\" (UniqueName: \"kubernetes.io/projected/e4cbed9b-e833-438e-b948-618ed3b525cf-kube-api-access-fvsrj\") pod \"e4cbed9b-e833-438e-b948-618ed3b525cf\" (UID: \"e4cbed9b-e833-438e-b948-618ed3b525cf\") "
Oct 30 20:46:57.274701 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:57.274648    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4cbed9b-e833-438e-b948-618ed3b525cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4cbed9b-e833-438e-b948-618ed3b525cf-metrics-certs\") pod \"e4cbed9b-e833-438e-b948-618ed3b525cf\" (UID: \"e4cbed9b-e833-438e-b948-618ed3b525cf\") "
Oct 30 20:46:57.274865 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:57.274795    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4cbed9b-e833-438e-b948-618ed3b525cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cbed9b-e833-438e-b948-618ed3b525cf-openshift-service-ca\") pod \"e4cbed9b-e833-438e-b948-618ed3b525cf\" (UID: \"e4cbed9b-e833-438e-b948-618ed3b525cf\") "
Oct 30 20:46:57.276824 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:57.276783    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4cbed9b-e833-438e-b948-618ed3b525cf-openshift-service-ca" pod "e4cbed9b-e833-438e-b948-618ed3b525cf" (UID: "e4cbed9b-e833-438e-b948-618ed3b525cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:46:57.305291 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:57.304537    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cbed9b-e833-438e-b948-618ed3b525cf-kube-api-access-fvsrj" pod "e4cbed9b-e833-438e-b948-618ed3b525cf" (UID: "e4cbed9b-e833-438e-b948-618ed3b525cf"). InnerVolumeSpecName "kube-api-access-fvsrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:46:57.305291 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:57.304537    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4cbed9b-e833-438e-b948-618ed3b525cf-metrics-certs" pod "e4cbed9b-e833-438e-b948-618ed3b525cf" (UID: "e4cbed9b-e833-438e-b948-618ed3b525cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:46:57.312612 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:57.309429    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4cbed9b-e833-438e-b948-618ed3b525cf-maas-api-tls" pod "e4cbed9b-e833-438e-b948-618ed3b525cf" (UID: "e4cbed9b-e833-438e-b948-618ed3b525cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:46:57.377223 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:57.377185    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvsrj\" (UniqueName: \"kubernetes.io/projected/e4cbed9b-e833-438e-b948-618ed3b525cf-kube-api-access-fvsrj\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:57.377223 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:57.377212    4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4cbed9b-e833-438e-b948-618ed3b525cf-metrics-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:57.377223 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:57.377222    4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cbed9b-e833-438e-b948-618ed3b525cf-openshift-service-ca\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:57.377223 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:57.377230    4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4cbed9b-e833-438e-b948-618ed3b525cf-maas-api-tls\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:57.576335 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:57.576286    4471 generic.go:356] "Generic (PLEG): container finished" podID="e4cbed9b-e833-438e-b948-618ed3b525cf" containerID="a465a2cc205b69a9789acbaf679bd8c5e46be65bcace8aaff8876397f4f0570b" exitCode=0
Oct 30 20:46:57.576507 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:57.576356    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr" event={"ID":"e4cbed9b-e833-438e-b948-618ed3b525cf","Type":"ContainerDied","Data":"a465a2cc205b69a9789acbaf679bd8c5e46be65bcace8aaff8876397f4f0570b"}
Oct 30 20:46:57.576507 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:57.576381    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"
Oct 30 20:46:57.576507 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:57.576419    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr" event={"ID":"e4cbed9b-e833-438e-b948-618ed3b525cf","Type":"ContainerDied","Data":"5e960703648e628f09a761926db1fc38f86e0cb58241134c5a74da5d2be07cb4"}
Oct 30 20:46:57.576507 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:57.576448    4471 scope.go:122] "RemoveContainer" containerID="a465a2cc205b69a9789acbaf679bd8c5e46be65bcace8aaff8876397f4f0570b"
Oct 30 20:46:57.586557 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:57.586539    4471 scope.go:122] "RemoveContainer" containerID="a465a2cc205b69a9789acbaf679bd8c5e46be65bcace8aaff8876397f4f0570b"
Oct 30 20:46:57.586822 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:46:57.586805    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a465a2cc205b69a9789acbaf679bd8c5e46be65bcace8aaff8876397f4f0570b\": container with ID starting with a465a2cc205b69a9789acbaf679bd8c5e46be65bcace8aaff8876397f4f0570b not found: ID does not exist" containerID="a465a2cc205b69a9789acbaf679bd8c5e46be65bcace8aaff8876397f4f0570b"
Oct 30 20:46:57.586892 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:57.586830    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a465a2cc205b69a9789acbaf679bd8c5e46be65bcace8aaff8876397f4f0570b"} err="failed to get container status \"a465a2cc205b69a9789acbaf679bd8c5e46be65bcace8aaff8876397f4f0570b\": rpc error: code = NotFound desc = could not find container \"a465a2cc205b69a9789acbaf679bd8c5e46be65bcace8aaff8876397f4f0570b\": container with ID starting with a465a2cc205b69a9789acbaf679bd8c5e46be65bcace8aaff8876397f4f0570b not found: ID does not exist"
Oct 30 20:46:57.741353 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:57.741237    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"]
Oct 30 20:46:57.802378 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:57.800668    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d757729-55c8b46766-gjktr"]
Oct 30 20:46:58.465345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:46:58.462983    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4cbed9b-e833-438e-b948-618ed3b525cf" path="/var/lib/kubelet/pods/e4cbed9b-e833-438e-b948-618ed3b525cf/volumes"
Oct 30 20:47:00.964727 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:47:00.964677    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:47:02.408270 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.402295    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"]
Oct 30 20:47:02.408270 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.403182    4471 cpu_manager.go:414] "containerMap: removing container" podUID="e4cbed9b-e833-438e-b948-618ed3b525cf" containerName="maas-api"
Oct 30 20:47:02.408270 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.403203    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cbed9b-e833-438e-b948-618ed3b525cf" containerName="maas-api"
Oct 30 20:47:02.408270 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.403389    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cbed9b-e833-438e-b948-618ed3b525cf" containerName="maas-api"
Oct 30 20:47:02.417031 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.410130    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.478831 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.478801    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"]
Oct 30 20:47:02.484342 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.483099    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b255dd-817d-480d-9f58-df362bca182d-istiod-ca-cert\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.484342 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.483169    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b255dd-817d-480d-9f58-df362bca182d-istio-podinfo\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.484342 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.483222    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-workload-certs\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.484342 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.483277    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-workload-socket\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.484342 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.483306    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b255dd-817d-480d-9f58-df362bca182d-istio-token\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.484342 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.483367    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6g66\" (UniqueName: \"kubernetes.io/projected/c7b255dd-817d-480d-9f58-df362bca182d-kube-api-access-x6g66\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.484342 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.483401    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-credential-socket\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.484342 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.483439    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-istio-data\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.484342 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.483584    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-istio-envoy\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.505592 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.505539    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-99c030-openshift-default-dockercfg-zv9p2\""
Oct 30 20:47:02.586652 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.586614    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-credential-socket\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.586919 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.586898    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-istio-data\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.587153 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.587133    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-istio-envoy\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.587363 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.587347    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b255dd-817d-480d-9f58-df362bca182d-istiod-ca-cert\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.587496 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.587480    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b255dd-817d-480d-9f58-df362bca182d-istio-podinfo\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.587647 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.587629    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-workload-certs\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.587797 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.587781    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-workload-socket\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.587912 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.587898    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b255dd-817d-480d-9f58-df362bca182d-istio-token\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.588101 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.588080    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6g66\" (UniqueName: \"kubernetes.io/projected/c7b255dd-817d-480d-9f58-df362bca182d-kube-api-access-x6g66\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.588982 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.588958    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-credential-socket\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.589396 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.589376    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-istio-data\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.590056 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.590034    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-workload-certs\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.592391 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.590808    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b255dd-817d-480d-9f58-df362bca182d-istiod-ca-cert\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.592527 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.591033    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-workload-socket\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.617056 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.613618    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-istio-envoy\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.658603 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.658519    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b255dd-817d-480d-9f58-df362bca182d-istio-podinfo\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.676030 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.675961    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b255dd-817d-480d-9f58-df362bca182d-istio-token\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.693349 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.688402    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6g66\" (UniqueName: \"kubernetes.io/projected/c7b255dd-817d-480d-9f58-df362bca182d-kube-api-access-x6g66\") pod \"e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") " pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:02.780345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:02.779596    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:03.250312 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:03.250269    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"]
Oct 30 20:47:03.251772 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:47:03.251746    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b255dd_817d_480d_9f58_df362bca182d.slice/crio-9867a2e4d709b417e6597ede78383b6a7629634cb54c66cab6284ed48ccb429b WatchSource:0}: Error finding container 9867a2e4d709b417e6597ede78383b6a7629634cb54c66cab6284ed48ccb429b: Status 404 returned error can't find the container with id 9867a2e4d709b417e6597ede78383b6a7629634cb54c66cab6284ed48ccb429b
Oct 30 20:47:03.255420 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:03.255378    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:47:03.255537 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:03.255496    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:47:03.255587 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:03.255542    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:47:03.732764 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:03.732725    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm" event={"ID":"c7b255dd-817d-480d-9f58-df362bca182d","Type":"ContainerStarted","Data":"2b4be582624ab4776007fa8c03a7350b7fa73b819fd32e2029bcd6b5f024b14c"}
Oct 30 20:47:03.733404 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:03.733376    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm" event={"ID":"c7b255dd-817d-480d-9f58-df362bca182d","Type":"ContainerStarted","Data":"9867a2e4d709b417e6597ede78383b6a7629634cb54c66cab6284ed48ccb429b"}
Oct 30 20:47:03.780119 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:03.780087    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:03.991424 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:03.990890    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm" podStartSLOduration=1.9908668550000002 podStartE2EDuration="1.990866855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:03.967700481 +0000 UTC m=+1643.646329095" watchObservedRunningTime="2026-10-30 20:47:03.990866855 +0000 UTC m=+1643.669495473"
Oct 30 20:47:04.787085 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:04.787054    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:05.743702 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:05.743674    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:05.744822 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:05.744801    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:47:28.105697 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:28.105617    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"]
Oct 30 20:47:28.106255 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:28.105844    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb" podUID="a7c2de2a-ca80-4191-af39-44f6d686a52e" containerName="istio-proxy" containerID="cri-o://fb14458278243aae1562baca64dd110d3f59f0e492ffd5e9ae6a1378a341ff56" gracePeriod=30
Oct 30 20:47:30.962841 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:47:30.962805    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:47:33.379412 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.379387    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:47:33.431192 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.431162    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-workload-socket\") pod \"a7c2de2a-ca80-4191-af39-44f6d686a52e\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") "
Oct 30 20:47:33.431362 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.431198    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-envoy\") pod \"a7c2de2a-ca80-4191-af39-44f6d686a52e\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") "
Oct 30 20:47:33.431362 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.431227    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-podinfo\") pod \"a7c2de2a-ca80-4191-af39-44f6d686a52e\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") "
Oct 30 20:47:33.431362 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.431267    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-data\") pod \"a7c2de2a-ca80-4191-af39-44f6d686a52e\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") "
Oct 30 20:47:33.431362 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.431349    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-credential-socket\") pod \"a7c2de2a-ca80-4191-af39-44f6d686a52e\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") "
Oct 30 20:47:33.431533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.431450    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c2de2a-ca80-4191-af39-44f6d686a52e-kube-api-access-4ftmw\" (UniqueName: \"kubernetes.io/projected/a7c2de2a-ca80-4191-af39-44f6d686a52e-kube-api-access-4ftmw\") pod \"a7c2de2a-ca80-4191-af39-44f6d686a52e\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") "
Oct 30 20:47:33.431533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.431477    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-workload-certs\") pod \"a7c2de2a-ca80-4191-af39-44f6d686a52e\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") "
Oct 30 20:47:33.431533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.431515    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-workload-socket" pod "a7c2de2a-ca80-4191-af39-44f6d686a52e" (UID: "a7c2de2a-ca80-4191-af39-44f6d686a52e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.431690 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.431563    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-token\" (UniqueName: \"kubernetes.io/projected/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-token\") pod \"a7c2de2a-ca80-4191-af39-44f6d686a52e\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") "
Oct 30 20:47:33.431690 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.431624    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c2de2a-ca80-4191-af39-44f6d686a52e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c2de2a-ca80-4191-af39-44f6d686a52e-istiod-ca-cert\") pod \"a7c2de2a-ca80-4191-af39-44f6d686a52e\" (UID: \"a7c2de2a-ca80-4191-af39-44f6d686a52e\") "
Oct 30 20:47:33.431792 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.431685    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-data" pod "a7c2de2a-ca80-4191-af39-44f6d686a52e" (UID: "a7c2de2a-ca80-4191-af39-44f6d686a52e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.431792 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.431705    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-credential-socket" pod "a7c2de2a-ca80-4191-af39-44f6d686a52e" (UID: "a7c2de2a-ca80-4191-af39-44f6d686a52e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.431896 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.431796    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-workload-certs" pod "a7c2de2a-ca80-4191-af39-44f6d686a52e" (UID: "a7c2de2a-ca80-4191-af39-44f6d686a52e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.432359 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.432303    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c2de2a-ca80-4191-af39-44f6d686a52e-istiod-ca-cert" pod "a7c2de2a-ca80-4191-af39-44f6d686a52e" (UID: "a7c2de2a-ca80-4191-af39-44f6d686a52e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:33.432359 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.432306    4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-workload-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.432707 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.432384    4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-data\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.432707 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.432402    4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-credential-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.432707 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.432419    4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-workload-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.433625 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.433604    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-envoy" pod "a7c2de2a-ca80-4191-af39-44f6d686a52e" (UID: "a7c2de2a-ca80-4191-af39-44f6d686a52e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:47:33.433729 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.433639    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-podinfo" pod "a7c2de2a-ca80-4191-af39-44f6d686a52e" (UID: "a7c2de2a-ca80-4191-af39-44f6d686a52e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:47:33.433729 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.433719    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c2de2a-ca80-4191-af39-44f6d686a52e-kube-api-access-4ftmw" pod "a7c2de2a-ca80-4191-af39-44f6d686a52e" (UID: "a7c2de2a-ca80-4191-af39-44f6d686a52e"). InnerVolumeSpecName "kube-api-access-4ftmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:33.434035 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.434019    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-token" pod "a7c2de2a-ca80-4191-af39-44f6d686a52e" (UID: "a7c2de2a-ca80-4191-af39-44f6d686a52e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:33.534034 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.533992    4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-envoy\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.534034 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.534035    4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-podinfo\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.534279 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.534052    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ftmw\" (UniqueName: \"kubernetes.io/projected/a7c2de2a-ca80-4191-af39-44f6d686a52e-kube-api-access-4ftmw\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.534279 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.534066    4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c2de2a-ca80-4191-af39-44f6d686a52e-istio-token\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.534279 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.534080    4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c2de2a-ca80-4191-af39-44f6d686a52e-istiod-ca-cert\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:33.963046 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.963012    4471 generic.go:356] "Generic (PLEG): container finished" podID="a7c2de2a-ca80-4191-af39-44f6d686a52e" containerID="fb14458278243aae1562baca64dd110d3f59f0e492ffd5e9ae6a1378a341ff56" exitCode=0
Oct 30 20:47:33.963230 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.963050    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb" event={"ID":"a7c2de2a-ca80-4191-af39-44f6d686a52e","Type":"ContainerDied","Data":"fb14458278243aae1562baca64dd110d3f59f0e492ffd5e9ae6a1378a341ff56"}
Oct 30 20:47:33.963230 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.963079    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"
Oct 30 20:47:33.963230 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.963111    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb" event={"ID":"a7c2de2a-ca80-4191-af39-44f6d686a52e","Type":"ContainerDied","Data":"06f6dc2dccb85ebeab00cd43386cd4f614da987828517a319299b2c2e4eb2588"}
Oct 30 20:47:33.963230 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.963139    4471 scope.go:122] "RemoveContainer" containerID="fb14458278243aae1562baca64dd110d3f59f0e492ffd5e9ae6a1378a341ff56"
Oct 30 20:47:33.974541 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.974522    4471 scope.go:122] "RemoveContainer" containerID="fb14458278243aae1562baca64dd110d3f59f0e492ffd5e9ae6a1378a341ff56"
Oct 30 20:47:33.974871 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:47:33.974850    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb14458278243aae1562baca64dd110d3f59f0e492ffd5e9ae6a1378a341ff56\": container with ID starting with fb14458278243aae1562baca64dd110d3f59f0e492ffd5e9ae6a1378a341ff56 not found: ID does not exist" containerID="fb14458278243aae1562baca64dd110d3f59f0e492ffd5e9ae6a1378a341ff56"
Oct 30 20:47:33.974946 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:33.974880    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb14458278243aae1562baca64dd110d3f59f0e492ffd5e9ae6a1378a341ff56"} err="failed to get container status \"fb14458278243aae1562baca64dd110d3f59f0e492ffd5e9ae6a1378a341ff56\": rpc error: code = NotFound desc = could not find container \"fb14458278243aae1562baca64dd110d3f59f0e492ffd5e9ae6a1378a341ff56\": container with ID starting with fb14458278243aae1562baca64dd110d3f59f0e492ffd5e9ae6a1378a341ff56 not found: ID does not exist"
Oct 30 20:47:34.132817 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:34.132787    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"]
Oct 30 20:47:34.181410 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:34.181374    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-qmrfb"]
Oct 30 20:47:34.432078 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:34.432041    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c2de2a-ca80-4191-af39-44f6d686a52e" path="/var/lib/kubelet/pods/a7c2de2a-ca80-4191-af39-44f6d686a52e/volumes"
Oct 30 20:47:53.127170 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:53.127112    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt"]
Oct 30 20:47:53.128636 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:53.128608    4471 cpu_manager.go:414] "containerMap: removing container" podUID="a7c2de2a-ca80-4191-af39-44f6d686a52e" containerName="istio-proxy"
Oct 30 20:47:53.128821 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:53.128804    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c2de2a-ca80-4191-af39-44f6d686a52e" containerName="istio-proxy"
Oct 30 20:47:53.129279 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:53.129259    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c2de2a-ca80-4191-af39-44f6d686a52e" containerName="istio-proxy"
Oct 30 20:47:53.164185 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:53.164153    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt"
Oct 30 20:47:53.190584 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:53.190420    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt"]
Oct 30 20:47:53.234896 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:53.234858    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-061bda\""
Oct 30 20:47:53.235101 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:53.234953    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-061bda-dockercfg-6bjqg\""
Oct 30 20:47:53.241744 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:53.241722    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93409cb5-1925-4012-8983-1491ead93a97-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt\" (UID: \"93409cb5-1925-4012-8983-1491ead93a97\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt"
Oct 30 20:47:53.241855 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:53.241767    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8twz\" (UniqueName: \"kubernetes.io/projected/93409cb5-1925-4012-8983-1491ead93a97-kube-api-access-k8twz\") pod \"payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt\" (UID: \"93409cb5-1925-4012-8983-1491ead93a97\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt"
Oct 30 20:47:53.342670 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:53.342638    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8twz\" (UniqueName: \"kubernetes.io/projected/93409cb5-1925-4012-8983-1491ead93a97-kube-api-access-k8twz\") pod \"payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt\" (UID: \"93409cb5-1925-4012-8983-1491ead93a97\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt"
Oct 30 20:47:53.342910 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:53.342891    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93409cb5-1925-4012-8983-1491ead93a97-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt\" (UID: \"93409cb5-1925-4012-8983-1491ead93a97\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt"
Oct 30 20:47:53.343661 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:53.343634    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93409cb5-1925-4012-8983-1491ead93a97-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt\" (UID: \"93409cb5-1925-4012-8983-1491ead93a97\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt"
Oct 30 20:47:53.399883 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:53.399814    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8twz\" (UniqueName: \"kubernetes.io/projected/93409cb5-1925-4012-8983-1491ead93a97-kube-api-access-k8twz\") pod \"payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt\" (UID: \"93409cb5-1925-4012-8983-1491ead93a97\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt"
Oct 30 20:47:53.494347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:53.491560    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt"
Oct 30 20:47:53.890300 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:53.890276    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt"]
Oct 30 20:47:53.891967 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:47:53.891933    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93409cb5_1925_4012_8983_1491ead93a97.slice/crio-989fa666df8dd1b86f2a2345eb10f16971a44cb307e9c7fbda9ddd82dfea840a WatchSource:0}: Error finding container 989fa666df8dd1b86f2a2345eb10f16971a44cb307e9c7fbda9ddd82dfea840a: Status 404 returned error can't find the container with id 989fa666df8dd1b86f2a2345eb10f16971a44cb307e9c7fbda9ddd82dfea840a
Oct 30 20:47:54.088815 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:54.088784    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt" event={"ID":"93409cb5-1925-4012-8983-1491ead93a97","Type":"ContainerStarted","Data":"1e470e5f528c8b77f4bdea4916d5a7d164423ab5f996cf73465e04500078793c"}
Oct 30 20:47:54.088971 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:54.088829    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt" event={"ID":"93409cb5-1925-4012-8983-1491ead93a97","Type":"ContainerStarted","Data":"989fa666df8dd1b86f2a2345eb10f16971a44cb307e9c7fbda9ddd82dfea840a"}
Oct 30 20:47:54.088971 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:54.088859    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt"
Oct 30 20:47:54.204293 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:47:54.204195    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt" podStartSLOduration=1.204180882 podStartE2EDuration="1.204180882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:54.20281712 +0000 UTC m=+1693.881445737" watchObservedRunningTime="2026-10-30 20:47:54.204180882 +0000 UTC m=+1693.882809498"
Oct 30 20:48:00.955733 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:48:00.955695    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:48:02.111593 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:02.111558    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"]
Oct 30 20:48:02.115422 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:02.115399    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"
Oct 30 20:48:02.170227 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:02.170202    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-0ac4e5-serving-cert\""
Oct 30 20:48:02.193192 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:02.193159    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g\" (UID: \"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"
Oct 30 20:48:02.193402 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:02.193256    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf4lq\" (UniqueName: \"kubernetes.io/projected/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-kube-api-access-sf4lq\") pod \"maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g\" (UID: \"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"
Oct 30 20:48:02.193402 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:02.193315    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g\" (UID: \"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"
Oct 30 20:48:02.193529 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:02.193507    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g\" (UID: \"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"
Oct 30 20:48:02.219582 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:02.219556    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"]
Oct 30 20:48:02.294373 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:02.294341    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g\" (UID: \"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"
Oct 30 20:48:02.294556 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:02.294429    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g\" (UID: \"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"
Oct 30 20:48:02.294556 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:48:02.294480    4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-serving-cert: secret "maas-api-e2e-shared-a-w5-0ac4e5-serving-cert" not found
Oct 30 20:48:02.294556 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:02.294490    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf4lq\" (UniqueName: \"kubernetes.io/projected/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-kube-api-access-sf4lq\") pod \"maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g\" (UID: \"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"
Oct 30 20:48:02.294556 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:48:02.294536    4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-maas-api-tls podName:0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9 nodeName:}" failed. No retries permitted until 2026-10-30 20:48:02.794521947 +0000 UTC m=+1702.473150570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-maas-api-tls") pod "maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g" (UID: "0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9") : secret "maas-api-e2e-shared-a-w5-0ac4e5-serving-cert" not found
Oct 30 20:48:02.294734 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:02.294582    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g\" (UID: \"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"
Oct 30 20:48:02.295237 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:02.295218    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g\" (UID: \"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"
Oct 30 20:48:02.296977 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:02.296956    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g\" (UID: \"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"
Oct 30 20:48:02.387346 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:02.386919    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf4lq\" (UniqueName: \"kubernetes.io/projected/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-kube-api-access-sf4lq\") pod \"maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g\" (UID: \"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"
Oct 30 20:48:02.805650 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:02.805567    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g\" (UID: \"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"
Oct 30 20:48:02.808031 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:02.808009    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g\" (UID: \"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"
Oct 30 20:48:03.032106 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:03.032072    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"
Oct 30 20:48:03.412867 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:03.412833    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"]
Oct 30 20:48:03.414693 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:48:03.414667    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad51c43_2b3f_4b2e_bc30_5e847bcad9f9.slice/crio-f996f35008fa774700ea101c19109378c7f76390324f6b3e304aa4ef36da4c20 WatchSource:0}: Error finding container f996f35008fa774700ea101c19109378c7f76390324f6b3e304aa4ef36da4c20: Status 404 returned error can't find the container with id f996f35008fa774700ea101c19109378c7f76390324f6b3e304aa4ef36da4c20
Oct 30 20:48:04.211919 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:04.211806    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g" event={"ID":"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9","Type":"ContainerStarted","Data":"f996f35008fa774700ea101c19109378c7f76390324f6b3e304aa4ef36da4c20"}
Oct 30 20:48:05.145714 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:05.144523    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt"
Oct 30 20:48:05.221940 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:05.221895    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g" event={"ID":"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9","Type":"ContainerStarted","Data":"e14690aca0b0165197ae643eb25ac11d55d13d2726ba985d35026e309474c056"}
Oct 30 20:48:05.222098 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:05.222042    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"
Oct 30 20:48:05.378374 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:05.378308    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g" podStartSLOduration=2.542430755 podStartE2EDuration="3.37829323s" totalImagesPullingTime="835.862475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:05.375457674 +0000 UTC m=+1705.054086296" watchObservedRunningTime="2026-10-30 20:48:05.37829323 +0000 UTC m=+1705.056921845"
Oct 30 20:48:10.127819 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:10.127786    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"]
Oct 30 20:48:10.128202 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:10.128031    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv" podUID="35507347-96f8-4fbe-ad4d-49e8ec22eaf7" containerName="istio-proxy" containerID="cri-o://4b393651c8f5cd7e2a583eea1b82bde6c78923f4a943a67969b646cbc3d5f284" gracePeriod=30
Oct 30 20:48:11.264725 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:11.264695    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"
Oct 30 20:48:15.316047 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.316017    4471 generic.go:356] "Generic (PLEG): container finished" podID="35507347-96f8-4fbe-ad4d-49e8ec22eaf7" containerID="4b393651c8f5cd7e2a583eea1b82bde6c78923f4a943a67969b646cbc3d5f284" exitCode=0
Oct 30 20:48:15.316466 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.316155    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv" event={"ID":"35507347-96f8-4fbe-ad4d-49e8ec22eaf7","Type":"ContainerDied","Data":"4b393651c8f5cd7e2a583eea1b82bde6c78923f4a943a67969b646cbc3d5f284"}
Oct 30 20:48:15.389707 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.389682    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:48:15.529515 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.529424    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-workload-certs\") pod \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") "
Oct 30 20:48:15.529515 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.529508    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-token\" (UniqueName: \"kubernetes.io/projected/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-token\") pod \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") "
Oct 30 20:48:15.529750 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.529557    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-workload-socket\") pod \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") "
Oct 30 20:48:15.529750 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.529636    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-credential-socket\") pod \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") "
Oct 30 20:48:15.529750 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.529700    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-envoy\") pod \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") "
Oct 30 20:48:15.529750 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.529727    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-workload-certs" pod "35507347-96f8-4fbe-ad4d-49e8ec22eaf7" (UID: "35507347-96f8-4fbe-ad4d-49e8ec22eaf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.530025 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.529768    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-data\") pod \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") "
Oct 30 20:48:15.530025 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.529873    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-kube-api-access-wsp2j\" (UniqueName: \"kubernetes.io/projected/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-kube-api-access-wsp2j\") pod \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") "
Oct 30 20:48:15.530025 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.529916    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-podinfo\") pod \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") "
Oct 30 20:48:15.530025 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.529962    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istiod-ca-cert\") pod \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\" (UID: \"35507347-96f8-4fbe-ad4d-49e8ec22eaf7\") "
Oct 30 20:48:15.530451 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.530429    4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-workload-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.530553 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.530530    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istiod-ca-cert" pod "35507347-96f8-4fbe-ad4d-49e8ec22eaf7" (UID: "35507347-96f8-4fbe-ad4d-49e8ec22eaf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:48:15.532624 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.532596    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-credential-socket" pod "35507347-96f8-4fbe-ad4d-49e8ec22eaf7" (UID: "35507347-96f8-4fbe-ad4d-49e8ec22eaf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.532624 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.532611    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-workload-socket" pod "35507347-96f8-4fbe-ad4d-49e8ec22eaf7" (UID: "35507347-96f8-4fbe-ad4d-49e8ec22eaf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.532782 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.532625    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-data" pod "35507347-96f8-4fbe-ad4d-49e8ec22eaf7" (UID: "35507347-96f8-4fbe-ad4d-49e8ec22eaf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.532841 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.532804    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-envoy" pod "35507347-96f8-4fbe-ad4d-49e8ec22eaf7" (UID: "35507347-96f8-4fbe-ad4d-49e8ec22eaf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:48:15.533039 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.533015    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-kube-api-access-wsp2j" pod "35507347-96f8-4fbe-ad4d-49e8ec22eaf7" (UID: "35507347-96f8-4fbe-ad4d-49e8ec22eaf7"). InnerVolumeSpecName "kube-api-access-wsp2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:48:15.533268 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.533244    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-podinfo" pod "35507347-96f8-4fbe-ad4d-49e8ec22eaf7" (UID: "35507347-96f8-4fbe-ad4d-49e8ec22eaf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:48:15.533410 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.533265    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-token" pod "35507347-96f8-4fbe-ad4d-49e8ec22eaf7" (UID: "35507347-96f8-4fbe-ad4d-49e8ec22eaf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:48:15.632153 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.632115    4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-workload-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.632153 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.632158    4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-credential-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.632459 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.632174    4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-envoy\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.632459 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.632188    4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-data\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.632459 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.632205    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsp2j\" (UniqueName: \"kubernetes.io/projected/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-kube-api-access-wsp2j\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.632459 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.632218    4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-podinfo\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.632459 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.632232    4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istiod-ca-cert\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:15.632459 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:15.632246    4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35507347-96f8-4fbe-ad4d-49e8ec22eaf7-istio-token\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:48:16.322856 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:16.322830    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"
Oct 30 20:48:16.323247 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:16.322855    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv" event={"ID":"35507347-96f8-4fbe-ad4d-49e8ec22eaf7","Type":"ContainerDied","Data":"4d4686bd0fc50aa414d909d4ee7a5d247f42cc3f52752a6ae5e0480aef4c7ca9"}
Oct 30 20:48:16.323247 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:16.322949    4471 scope.go:122] "RemoveContainer" containerID="4b393651c8f5cd7e2a583eea1b82bde6c78923f4a943a67969b646cbc3d5f284"
Oct 30 20:48:16.542225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:16.542191    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"]
Oct 30 20:48:16.591705 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:16.591634    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-89wnv"]
Oct 30 20:48:18.432045 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:18.432009    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35507347-96f8-4fbe-ad4d-49e8ec22eaf7" path="/var/lib/kubelet/pods/35507347-96f8-4fbe-ad4d-49e8ec22eaf7/volumes"
Oct 30 20:48:29.104948 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.104911    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"]
Oct 30 20:48:29.105477 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.105463    4471 cpu_manager.go:414] "containerMap: removing container" podUID="35507347-96f8-4fbe-ad4d-49e8ec22eaf7" containerName="istio-proxy"
Oct 30 20:48:29.105522 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.105478    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35507347-96f8-4fbe-ad4d-49e8ec22eaf7" containerName="istio-proxy"
Oct 30 20:48:29.105586 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.105578    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="35507347-96f8-4fbe-ad4d-49e8ec22eaf7" containerName="istio-proxy"
Oct 30 20:48:29.110197 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.110179    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:29.169566 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.169531    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"]
Oct 30 20:48:29.189577 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.189550    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-27219c73\"/\"openshift-service-ca.crt\""
Oct 30 20:48:29.189577 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.189569    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-27219c73\"/\"kube-root-ca.crt\""
Oct 30 20:48:29.202178 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.202145    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-dshm\") pod \"e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") " pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:29.202373 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.202220    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-tmp-dir\") pod \"e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") " pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:29.202448 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.202388    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-home\") pod \"e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") " pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:29.202448 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.202416    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-model-cache\") pod \"e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") " pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:29.202448 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.202439    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpj64\" (UniqueName: \"kubernetes.io/projected/ee5d26da-85dc-441e-86ac-87ea9512b8c5-kube-api-access-tpj64\") pod \"e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") " pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:29.202614 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.202468    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee5d26da-85dc-441e-86ac-87ea9512b8c5-tls-certs\") pod \"e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") " pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:29.214133 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.214106    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-27219c73\"/\"default-dockercfg-j5b8m\""
Oct 30 20:48:29.214261 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.214168    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-27219c73\"/\"e2e-lifecycle-model-27219c73-kserve-self-signed-certs\""
Oct 30 20:48:29.314351 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.311711    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-home\") pod \"e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") " pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:29.314351 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.311773    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-model-cache\") pod \"e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") " pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:29.314351 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.311812    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpj64\" (UniqueName: \"kubernetes.io/projected/ee5d26da-85dc-441e-86ac-87ea9512b8c5-kube-api-access-tpj64\") pod \"e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") " pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:29.314351 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.311849    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee5d26da-85dc-441e-86ac-87ea9512b8c5-tls-certs\") pod \"e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") " pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:29.314351 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.311972    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-dshm\") pod \"e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") " pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:29.314351 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.312080    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-tmp-dir\") pod \"e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") " pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:29.314351 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.312821    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-tmp-dir\") pod \"e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") " pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:29.314351 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.313109    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-home\") pod \"e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") " pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:29.314351 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.313415    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-model-cache\") pod \"e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") " pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:29.343341 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.341116    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee5d26da-85dc-441e-86ac-87ea9512b8c5-tls-certs\") pod \"e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") " pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:29.359634 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.357163    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-dshm\") pod \"e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") " pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:29.370538 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.370511    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpj64\" (UniqueName: \"kubernetes.io/projected/ee5d26da-85dc-441e-86ac-87ea9512b8c5-kube-api-access-tpj64\") pod \"e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") " pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:29.422277 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.422242    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:29.824692 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:29.824664    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"]
Oct 30 20:48:29.826111 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:48:29.826082    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee5d26da_85dc_441e_86ac_87ea9512b8c5.slice/crio-d99ae266bc8d84fbb9ed53390d8ea303c4cd75074623b52e84503bc63d343889 WatchSource:0}: Error finding container d99ae266bc8d84fbb9ed53390d8ea303c4cd75074623b52e84503bc63d343889: Status 404 returned error can't find the container with id d99ae266bc8d84fbb9ed53390d8ea303c4cd75074623b52e84503bc63d343889
Oct 30 20:48:30.431875 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:30.431836    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7" event={"ID":"ee5d26da-85dc-441e-86ac-87ea9512b8c5","Type":"ContainerStarted","Data":"b8de4c27843359ab8efdec21ca723ff4c2f6c0930479de1a213cdb4286680a29"}
Oct 30 20:48:30.431875 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:30.431878    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7" event={"ID":"ee5d26da-85dc-441e-86ac-87ea9512b8c5","Type":"ContainerStarted","Data":"d99ae266bc8d84fbb9ed53390d8ea303c4cd75074623b52e84503bc63d343889"}
Oct 30 20:48:30.557580 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:30.557530    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7" podStartSLOduration=1.557515762 podStartE2EDuration="1.557515762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:30.557011192 +0000 UTC m=+1730.235639808" watchObservedRunningTime="2026-10-30 20:48:30.557515762 +0000 UTC m=+1730.236144762"
Oct 30 20:48:30.954608 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:48:30.954572    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:48:31.422982 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:31.422947    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:39.423485 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:39.423448    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:39.436500 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:39.436474    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:39.492479 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:39.492451    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:48:44.109774 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:44.109721    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh" podUID="3dcbb5f5-93f4-4210-8274-2278fb9924ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:44.120343 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:44.118640    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf" podUID="bf89b9b8-0419-48bc-b410-12807dab75db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:44.120902 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:44.120569    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b" podUID="1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:44.302645 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:44.302590    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7" podUID="02b9bb13-c775-44a5-b034-563359097687" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 20:48:44.348611 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:44.348565    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk" podUID="a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:44.540914 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:44.540847    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht" podUID="5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:44.576366 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:44.575429    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl" podUID="a4344406-c869-420f-8cf3-93c5e0549be8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:44.621493 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:44.621446    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524" podUID="31cc010d-9e1f-42c2-a736-0ceacbedb8c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded"
Oct 30 20:48:51.784502 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:51.784462    4471 patch_prober.go:29] interesting pod/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:48:51.784931 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:51.784536    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm" podUID="c7b255dd-817d-480d-9f58-df362bca182d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:48:52.622238 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:52.622136    4471 generic.go:356] "Generic (PLEG): container finished" podID="c7b255dd-817d-480d-9f58-df362bca182d" containerID="2b4be582624ab4776007fa8c03a7350b7fa73b819fd32e2029bcd6b5f024b14c" exitCode=0
Oct 30 20:48:52.622238 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:52.622209    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm" event={"ID":"c7b255dd-817d-480d-9f58-df362bca182d","Type":"ContainerDied","Data":"2b4be582624ab4776007fa8c03a7350b7fa73b819fd32e2029bcd6b5f024b14c"}
Oct 30 20:48:52.622690 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:52.622672    4471 scope.go:122] "RemoveContainer" containerID="2b4be582624ab4776007fa8c03a7350b7fa73b819fd32e2029bcd6b5f024b14c"
Oct 30 20:48:52.624305 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:52.624277    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:48:52.624413 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:52.624366    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:48:52.624413 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:52.624401    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:48:53.630886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:53.630851    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm" event={"ID":"c7b255dd-817d-480d-9f58-df362bca182d","Type":"ContainerStarted","Data":"16218d06bc5d19172fc894ba0fdb105c0914a0820087a85fd54a876faec2a3ad"}
Oct 30 20:48:53.780871 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:53.780830    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:48:53.781105 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:53.781082    4471 patch_prober.go:29] interesting pod/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 20:48:53.781167 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:53.781135    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm" podUID="c7b255dd-817d-480d-9f58-df362bca182d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 20:48:54.780930 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:54.780889    4471 patch_prober.go:29] interesting pod/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 20:48:54.781372 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:54.780942    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm" podUID="c7b255dd-817d-480d-9f58-df362bca182d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 20:48:55.780705 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:55.780669    4471 patch_prober.go:29] interesting pod/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 20:48:55.780893 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:55.780725    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm" podUID="c7b255dd-817d-480d-9f58-df362bca182d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 20:48:56.780903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:56.780857    4471 patch_prober.go:29] interesting pod/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 20:48:56.781347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:56.780927    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm" podUID="c7b255dd-817d-480d-9f58-df362bca182d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 20:48:57.780303 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:57.780263    4471 patch_prober.go:29] interesting pod/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 20:48:57.780526 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:57.780365    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm" podUID="c7b255dd-817d-480d-9f58-df362bca182d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 20:48:58.780919 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:58.780886    4471 patch_prober.go:29] interesting pod/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 20:48:58.781315 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:58.780938    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm" podUID="c7b255dd-817d-480d-9f58-df362bca182d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 20:48:59.784779 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:59.784750    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:48:59.785279 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:59.785244    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:48:59.786068 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:48:59.786049    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:49:00.953817 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:49:00.953774    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:49:19.076347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:19.075721    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l"]
Oct 30 20:49:24.076978 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:24.076925    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l" podUID="b6133567-cd29-4080-a47f-72ce483242c9" containerName="payload-pre-processing" containerID="cri-o://ead845c7571ef9b1e38253d859f5782492dc139d2233d8079a15f2da88499410" gracePeriod=25
Oct 30 20:49:24.354840 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:24.354810    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l"
Oct 30 20:49:24.385366 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:24.385336    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6133567-cd29-4080-a47f-72ce483242c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6133567-cd29-4080-a47f-72ce483242c9-plugins-config-volume\") pod \"b6133567-cd29-4080-a47f-72ce483242c9\" (UID: \"b6133567-cd29-4080-a47f-72ce483242c9\") "
Oct 30 20:49:24.385514 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:24.385485    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6133567-cd29-4080-a47f-72ce483242c9-kube-api-access-j9fx7\" (UniqueName: \"kubernetes.io/projected/b6133567-cd29-4080-a47f-72ce483242c9-kube-api-access-j9fx7\") pod \"b6133567-cd29-4080-a47f-72ce483242c9\" (UID: \"b6133567-cd29-4080-a47f-72ce483242c9\") "
Oct 30 20:49:24.385689 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:24.385664    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6133567-cd29-4080-a47f-72ce483242c9-plugins-config-volume" pod "b6133567-cd29-4080-a47f-72ce483242c9" (UID: "b6133567-cd29-4080-a47f-72ce483242c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:49:24.385844 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:24.385831    4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6133567-cd29-4080-a47f-72ce483242c9-plugins-config-volume\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:49:24.387672 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:24.387650    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6133567-cd29-4080-a47f-72ce483242c9-kube-api-access-j9fx7" pod "b6133567-cd29-4080-a47f-72ce483242c9" (UID: "b6133567-cd29-4080-a47f-72ce483242c9"). InnerVolumeSpecName "kube-api-access-j9fx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:49:24.487095 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:24.487066    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9fx7\" (UniqueName: \"kubernetes.io/projected/b6133567-cd29-4080-a47f-72ce483242c9-kube-api-access-j9fx7\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:49:24.906453 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:24.906414    4471 generic.go:356] "Generic (PLEG): container finished" podID="b6133567-cd29-4080-a47f-72ce483242c9" containerID="ead845c7571ef9b1e38253d859f5782492dc139d2233d8079a15f2da88499410" exitCode=0
Oct 30 20:49:24.906658 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:24.906457    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l" event={"ID":"b6133567-cd29-4080-a47f-72ce483242c9","Type":"ContainerDied","Data":"ead845c7571ef9b1e38253d859f5782492dc139d2233d8079a15f2da88499410"}
Oct 30 20:49:24.906658 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:24.906465    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l"
Oct 30 20:49:24.906658 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:24.906509    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l" event={"ID":"b6133567-cd29-4080-a47f-72ce483242c9","Type":"ContainerDied","Data":"1d9a0c90ffa1a07c7c3d2670b004759f69a3b16e7722756139e4151187397287"}
Oct 30 20:49:24.906658 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:24.906536    4471 scope.go:122] "RemoveContainer" containerID="ead845c7571ef9b1e38253d859f5782492dc139d2233d8079a15f2da88499410"
Oct 30 20:49:24.943219 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:24.943190    4471 scope.go:122] "RemoveContainer" containerID="ead845c7571ef9b1e38253d859f5782492dc139d2233d8079a15f2da88499410"
Oct 30 20:49:24.943806 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:49:24.943780    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ead845c7571ef9b1e38253d859f5782492dc139d2233d8079a15f2da88499410\": container with ID starting with ead845c7571ef9b1e38253d859f5782492dc139d2233d8079a15f2da88499410 not found: ID does not exist" containerID="ead845c7571ef9b1e38253d859f5782492dc139d2233d8079a15f2da88499410"
Oct 30 20:49:24.943964 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:24.943937    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ead845c7571ef9b1e38253d859f5782492dc139d2233d8079a15f2da88499410"} err="failed to get container status \"ead845c7571ef9b1e38253d859f5782492dc139d2233d8079a15f2da88499410\": rpc error: code = NotFound desc = could not find container \"ead845c7571ef9b1e38253d859f5782492dc139d2233d8079a15f2da88499410\": container with ID starting with ead845c7571ef9b1e38253d859f5782492dc139d2233d8079a15f2da88499410 not found: ID does not exist"
Oct 30 20:49:25.082378 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:25.082345    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l"]
Oct 30 20:49:25.106706 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:25.106673    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c650f2b7-75749c78cf-qpg7l"]
Oct 30 20:49:26.432340 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:26.432300    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6133567-cd29-4080-a47f-72ce483242c9" path="/var/lib/kubelet/pods/b6133567-cd29-4080-a47f-72ce483242c9/volumes"
Oct 30 20:49:30.955579 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:49:30.955542    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:49:35.364460 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:35.364372    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6"]
Oct 30 20:49:35.365986 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:35.365953    4471 cpu_manager.go:414] "containerMap: removing container" podUID="b6133567-cd29-4080-a47f-72ce483242c9" containerName="payload-pre-processing"
Oct 30 20:49:35.366169 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:35.366155    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6133567-cd29-4080-a47f-72ce483242c9" containerName="payload-pre-processing"
Oct 30 20:49:35.366423 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:35.366409    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6133567-cd29-4080-a47f-72ce483242c9" containerName="payload-pre-processing"
Oct 30 20:49:35.369218 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:35.369201    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6"
Oct 30 20:49:35.371597 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:35.371516    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6"]
Oct 30 20:49:35.373949 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:35.373917    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xztmr\" (UniqueName: \"kubernetes.io/projected/e9dd5e0a-7d52-4c72-a274-348b97f5dcf1-kube-api-access-xztmr\") pod \"payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6\" (UID: \"e9dd5e0a-7d52-4c72-a274-348b97f5dcf1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6"
Oct 30 20:49:35.374063 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:35.373980    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9dd5e0a-7d52-4c72-a274-348b97f5dcf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6\" (UID: \"e9dd5e0a-7d52-4c72-a274-348b97f5dcf1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6"
Oct 30 20:49:35.470728 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:35.470689    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-ab9c38\""
Oct 30 20:49:35.470728 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:35.470689    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-ab9c38-dockercfg-r55lx\""
Oct 30 20:49:35.475218 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:35.475191    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xztmr\" (UniqueName: \"kubernetes.io/projected/e9dd5e0a-7d52-4c72-a274-348b97f5dcf1-kube-api-access-xztmr\") pod \"payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6\" (UID: \"e9dd5e0a-7d52-4c72-a274-348b97f5dcf1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6"
Oct 30 20:49:35.475316 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:35.475235    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9dd5e0a-7d52-4c72-a274-348b97f5dcf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6\" (UID: \"e9dd5e0a-7d52-4c72-a274-348b97f5dcf1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6"
Oct 30 20:49:35.476038 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:35.476020    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9dd5e0a-7d52-4c72-a274-348b97f5dcf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6\" (UID: \"e9dd5e0a-7d52-4c72-a274-348b97f5dcf1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6"
Oct 30 20:49:35.532197 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:35.532159    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xztmr\" (UniqueName: \"kubernetes.io/projected/e9dd5e0a-7d52-4c72-a274-348b97f5dcf1-kube-api-access-xztmr\") pod \"payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6\" (UID: \"e9dd5e0a-7d52-4c72-a274-348b97f5dcf1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6"
Oct 30 20:49:35.684486 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:35.684456    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6"
Oct 30 20:49:36.059497 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:36.059468    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6"]
Oct 30 20:49:36.060639 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:49:36.060615    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9dd5e0a_7d52_4c72_a274_348b97f5dcf1.slice/crio-144137d827b690e1e7807187146c7eb7b03dba56a8e1557b464a1957fb6f317b WatchSource:0}: Error finding container 144137d827b690e1e7807187146c7eb7b03dba56a8e1557b464a1957fb6f317b: Status 404 returned error can't find the container with id 144137d827b690e1e7807187146c7eb7b03dba56a8e1557b464a1957fb6f317b
Oct 30 20:49:37.040367 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:37.040312    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6" event={"ID":"e9dd5e0a-7d52-4c72-a274-348b97f5dcf1","Type":"ContainerStarted","Data":"c4e93d428f1cb035b3726de430351fa9678a6918d9ce55f25197efcbd5990037"}
Oct 30 20:49:37.040839 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:37.040385    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6" event={"ID":"e9dd5e0a-7d52-4c72-a274-348b97f5dcf1","Type":"ContainerStarted","Data":"144137d827b690e1e7807187146c7eb7b03dba56a8e1557b464a1957fb6f317b"}
Oct 30 20:49:37.040839 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:37.040554    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6"
Oct 30 20:49:37.166924 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:37.166859    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6" podStartSLOduration=2.166842917 podStartE2EDuration="2.166842917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:37.165036701 +0000 UTC m=+1796.843665320" watchObservedRunningTime="2026-10-30 20:49:37.166842917 +0000 UTC m=+1796.845471529"
Oct 30 20:49:40.248294 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.241827    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx"]
Oct 30 20:49:40.248294 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.246746    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx"
Oct 30 20:49:40.322651 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.322607    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx"]
Oct 30 20:49:40.362197 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.362164    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-f8ac60\""
Oct 30 20:49:40.362642 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.362282    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-f8ac60-dockercfg-t8c42\""
Oct 30 20:49:40.380289 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.380252    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfh4w\" (UniqueName: \"kubernetes.io/projected/c121a634-e346-4be2-872d-fc943d82ad37-kube-api-access-tfh4w\") pod \"payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx\" (UID: \"c121a634-e346-4be2-872d-fc943d82ad37\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx"
Oct 30 20:49:40.380498 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.380406    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c121a634-e346-4be2-872d-fc943d82ad37-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx\" (UID: \"c121a634-e346-4be2-872d-fc943d82ad37\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx"
Oct 30 20:49:40.415221 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.415177    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2"]
Oct 30 20:49:40.418459 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.418437    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2"
Oct 30 20:49:40.467115 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.467074    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2"]
Oct 30 20:49:40.487695 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.487653    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2\" (UID: \"e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2"
Oct 30 20:49:40.487843 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.487785    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpccx\" (UniqueName: \"kubernetes.io/projected/e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12-kube-api-access-fpccx\") pod \"payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2\" (UID: \"e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2"
Oct 30 20:49:40.487885 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.487853    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfh4w\" (UniqueName: \"kubernetes.io/projected/c121a634-e346-4be2-872d-fc943d82ad37-kube-api-access-tfh4w\") pod \"payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx\" (UID: \"c121a634-e346-4be2-872d-fc943d82ad37\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx"
Oct 30 20:49:40.488003 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.487987    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c121a634-e346-4be2-872d-fc943d82ad37-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx\" (UID: \"c121a634-e346-4be2-872d-fc943d82ad37\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx"
Oct 30 20:49:40.539002 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.538911    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-f8ac60\""
Oct 30 20:49:40.544680 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.544655    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfh4w\" (UniqueName: \"kubernetes.io/projected/c121a634-e346-4be2-872d-fc943d82ad37-kube-api-access-tfh4w\") pod \"payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx\" (UID: \"c121a634-e346-4be2-872d-fc943d82ad37\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx"
Oct 30 20:49:40.548940 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.548916    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c121a634-e346-4be2-872d-fc943d82ad37-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx\" (UID: \"c121a634-e346-4be2-872d-fc943d82ad37\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx"
Oct 30 20:49:40.589669 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.589641    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2\" (UID: \"e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2"
Oct 30 20:49:40.589786 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.589685    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpccx\" (UniqueName: \"kubernetes.io/projected/e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12-kube-api-access-fpccx\") pod \"payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2\" (UID: \"e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2"
Oct 30 20:49:40.590401 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.590382    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2\" (UID: \"e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2"
Oct 30 20:49:40.648534 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.648498    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpccx\" (UniqueName: \"kubernetes.io/projected/e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12-kube-api-access-fpccx\") pod \"payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2\" (UID: \"e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2"
Oct 30 20:49:40.683569 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.683541    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-f8ac60-dockercfg-t8c42\""
Oct 30 20:49:40.693011 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.692988    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx"
Oct 30 20:49:40.733690 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:40.733653    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2"
Oct 30 20:49:41.073684 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:41.073595    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx"]
Oct 30 20:49:41.127554 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:41.127524    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2"]
Oct 30 20:49:41.133208 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:49:41.133183    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e339ad_9117_4d8d_a11e_c9d2fd7ebb12.slice/crio-579fcc078c29854514271c20409f70c13d34f707503fc8872d6612167b177a43 WatchSource:0}: Error finding container 579fcc078c29854514271c20409f70c13d34f707503fc8872d6612167b177a43: Status 404 returned error can't find the container with id 579fcc078c29854514271c20409f70c13d34f707503fc8872d6612167b177a43
Oct 30 20:49:42.084505 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:42.084466    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx" event={"ID":"c121a634-e346-4be2-872d-fc943d82ad37","Type":"ContainerStarted","Data":"ff6e98c71b7a43006361600d34f2fd6d2aba47c745eed1889664f830ae8cfc14"}
Oct 30 20:49:42.085045 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:42.084530    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx" event={"ID":"c121a634-e346-4be2-872d-fc943d82ad37","Type":"ContainerStarted","Data":"58e28e333e2f31990f81a1cbc5633e222e1adfed8914809b3c5ec26ec7fcf7ef"}
Oct 30 20:49:42.085045 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:42.084562    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx"
Oct 30 20:49:42.086152 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:42.086112    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2" event={"ID":"e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12","Type":"ContainerStarted","Data":"97e2f89a97cb9b83888a7f649893bfafc686db82d9ab9a1b665a5bf2fb3d0f2c"}
Oct 30 20:49:42.086274 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:42.086168    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2" event={"ID":"e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12","Type":"ContainerStarted","Data":"579fcc078c29854514271c20409f70c13d34f707503fc8872d6612167b177a43"}
Oct 30 20:49:42.086274 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:42.086214    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2"
Oct 30 20:49:42.216748 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:42.216699    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx" podStartSLOduration=2.216684441 podStartE2EDuration="2.216684441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:42.211255462 +0000 UTC m=+1801.889884078" watchObservedRunningTime="2026-10-30 20:49:42.216684441 +0000 UTC m=+1801.895313057"
Oct 30 20:49:42.378377 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:42.374821    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2" podStartSLOduration=2.374797644 podStartE2EDuration="2.374797644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:42.354786005 +0000 UTC m=+1802.033414623" watchObservedRunningTime="2026-10-30 20:49:42.374797644 +0000 UTC m=+1802.053426261"
Oct 30 20:49:47.482285 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:47.482243    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"]
Oct 30 20:49:47.482808 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:47.482568    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt" podUID="fce24575-c3b1-4a12-a5be-43ead65afa74" containerName="maas-api" containerID="cri-o://0d01a065737520cf0b1d7bba6592cc51bf9d4f537a2eaa7b430658317f163c67" gracePeriod=30
Oct 30 20:49:48.057055 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.057009    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6"
Oct 30 20:49:48.063125 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.063048    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"
Oct 30 20:49:48.122857 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.122823    4471 generic.go:356] "Generic (PLEG): container finished" podID="fce24575-c3b1-4a12-a5be-43ead65afa74" containerID="0d01a065737520cf0b1d7bba6592cc51bf9d4f537a2eaa7b430658317f163c67" exitCode=0
Oct 30 20:49:48.123059 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.122917    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt" event={"ID":"fce24575-c3b1-4a12-a5be-43ead65afa74","Type":"ContainerDied","Data":"0d01a065737520cf0b1d7bba6592cc51bf9d4f537a2eaa7b430658317f163c67"}
Oct 30 20:49:48.123059 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.122977    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt" event={"ID":"fce24575-c3b1-4a12-a5be-43ead65afa74","Type":"ContainerDied","Data":"a1d1ce87643e922dae330af654d5c7cf3e19db58039a0df9aa8f0894cd8e401f"}
Oct 30 20:49:48.123059 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.123000    4471 scope.go:122] "RemoveContainer" containerID="0d01a065737520cf0b1d7bba6592cc51bf9d4f537a2eaa7b430658317f163c67"
Oct 30 20:49:48.123368 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.123315    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"
Oct 30 20:49:48.135115 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.135084    4471 scope.go:122] "RemoveContainer" containerID="0d01a065737520cf0b1d7bba6592cc51bf9d4f537a2eaa7b430658317f163c67"
Oct 30 20:49:48.135486 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:49:48.135456    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d01a065737520cf0b1d7bba6592cc51bf9d4f537a2eaa7b430658317f163c67\": container with ID starting with 0d01a065737520cf0b1d7bba6592cc51bf9d4f537a2eaa7b430658317f163c67 not found: ID does not exist" containerID="0d01a065737520cf0b1d7bba6592cc51bf9d4f537a2eaa7b430658317f163c67"
Oct 30 20:49:48.135592 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.135496    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d01a065737520cf0b1d7bba6592cc51bf9d4f537a2eaa7b430658317f163c67"} err="failed to get container status \"0d01a065737520cf0b1d7bba6592cc51bf9d4f537a2eaa7b430658317f163c67\": rpc error: code = NotFound desc = could not find container \"0d01a065737520cf0b1d7bba6592cc51bf9d4f537a2eaa7b430658317f163c67\": container with ID starting with 0d01a065737520cf0b1d7bba6592cc51bf9d4f537a2eaa7b430658317f163c67 not found: ID does not exist"
Oct 30 20:49:48.142065 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.142042    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fce24575-c3b1-4a12-a5be-43ead65afa74-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce24575-c3b1-4a12-a5be-43ead65afa74-openshift-service-ca\") pod \"fce24575-c3b1-4a12-a5be-43ead65afa74\" (UID: \"fce24575-c3b1-4a12-a5be-43ead65afa74\") "
Oct 30 20:49:48.142153 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.142092    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fce24575-c3b1-4a12-a5be-43ead65afa74-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fce24575-c3b1-4a12-a5be-43ead65afa74-maas-api-tls\") pod \"fce24575-c3b1-4a12-a5be-43ead65afa74\" (UID: \"fce24575-c3b1-4a12-a5be-43ead65afa74\") "
Oct 30 20:49:48.142281 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.142264    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fce24575-c3b1-4a12-a5be-43ead65afa74-kube-api-access-vbktw\" (UniqueName: \"kubernetes.io/projected/fce24575-c3b1-4a12-a5be-43ead65afa74-kube-api-access-vbktw\") pod \"fce24575-c3b1-4a12-a5be-43ead65afa74\" (UID: \"fce24575-c3b1-4a12-a5be-43ead65afa74\") "
Oct 30 20:49:48.142346 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.142304    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fce24575-c3b1-4a12-a5be-43ead65afa74-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fce24575-c3b1-4a12-a5be-43ead65afa74-metrics-certs\") pod \"fce24575-c3b1-4a12-a5be-43ead65afa74\" (UID: \"fce24575-c3b1-4a12-a5be-43ead65afa74\") "
Oct 30 20:49:48.142435 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.142393    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fce24575-c3b1-4a12-a5be-43ead65afa74-openshift-service-ca" pod "fce24575-c3b1-4a12-a5be-43ead65afa74" (UID: "fce24575-c3b1-4a12-a5be-43ead65afa74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:49:48.142797 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.142767    4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce24575-c3b1-4a12-a5be-43ead65afa74-openshift-service-ca\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:49:48.144520 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.144499    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fce24575-c3b1-4a12-a5be-43ead65afa74-maas-api-tls" pod "fce24575-c3b1-4a12-a5be-43ead65afa74" (UID: "fce24575-c3b1-4a12-a5be-43ead65afa74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:49:48.144625 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.144521    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fce24575-c3b1-4a12-a5be-43ead65afa74-metrics-certs" pod "fce24575-c3b1-4a12-a5be-43ead65afa74" (UID: "fce24575-c3b1-4a12-a5be-43ead65afa74"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:49:48.144625 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.144528    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce24575-c3b1-4a12-a5be-43ead65afa74-kube-api-access-vbktw" pod "fce24575-c3b1-4a12-a5be-43ead65afa74" (UID: "fce24575-c3b1-4a12-a5be-43ead65afa74"). InnerVolumeSpecName "kube-api-access-vbktw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:49:48.244042 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.243968    4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fce24575-c3b1-4a12-a5be-43ead65afa74-maas-api-tls\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:49:48.244042 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.243998    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbktw\" (UniqueName: \"kubernetes.io/projected/fce24575-c3b1-4a12-a5be-43ead65afa74-kube-api-access-vbktw\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:49:48.244042 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.244009    4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fce24575-c3b1-4a12-a5be-43ead65afa74-metrics-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:49:48.566845 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.566766    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"]
Oct 30 20:49:48.592345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:48.586152    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c650f2b7-fd54d8bf-zhlnt"]
Oct 30 20:49:50.433583 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:50.433551    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fce24575-c3b1-4a12-a5be-43ead65afa74" path="/var/lib/kubelet/pods/fce24575-c3b1-4a12-a5be-43ead65afa74/volumes"
Oct 30 20:49:53.110367 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:53.109704    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx"
Oct 30 20:49:53.131346 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:49:53.111227    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2"
Oct 30 20:50:00.964623 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:50:00.964584    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:50:07.838992 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:07.838962    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg"]
Oct 30 20:50:07.839511 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:07.839497    4471 cpu_manager.go:414] "containerMap: removing container" podUID="fce24575-c3b1-4a12-a5be-43ead65afa74" containerName="maas-api"
Oct 30 20:50:07.839564 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:07.839512    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fce24575-c3b1-4a12-a5be-43ead65afa74" containerName="maas-api"
Oct 30 20:50:07.839632 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:07.839623    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="fce24575-c3b1-4a12-a5be-43ead65afa74" containerName="maas-api"
Oct 30 20:50:07.842742 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:07.842726    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg"
Oct 30 20:50:07.879415 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:07.879366    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whmpq\" (UniqueName: \"kubernetes.io/projected/2a7a1c0a-2a90-487e-9a94-780475a94fb5-kube-api-access-whmpq\") pod \"payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg\" (UID: \"2a7a1c0a-2a90-487e-9a94-780475a94fb5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg"
Oct 30 20:50:07.879635 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:07.879617    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7a1c0a-2a90-487e-9a94-780475a94fb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg\" (UID: \"2a7a1c0a-2a90-487e-9a94-780475a94fb5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg"
Oct 30 20:50:07.909349 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:07.908359    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c3d13a-dockercfg-knsm6\""
Oct 30 20:50:07.909349 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:07.908537    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c3d13a\""
Oct 30 20:50:07.975349 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:07.975039    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg"]
Oct 30 20:50:07.989386 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:07.987162    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7a1c0a-2a90-487e-9a94-780475a94fb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg\" (UID: \"2a7a1c0a-2a90-487e-9a94-780475a94fb5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg"
Oct 30 20:50:07.989386 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:07.987237    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whmpq\" (UniqueName: \"kubernetes.io/projected/2a7a1c0a-2a90-487e-9a94-780475a94fb5-kube-api-access-whmpq\") pod \"payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg\" (UID: \"2a7a1c0a-2a90-487e-9a94-780475a94fb5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg"
Oct 30 20:50:07.989386 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:07.988232    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7a1c0a-2a90-487e-9a94-780475a94fb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg\" (UID: \"2a7a1c0a-2a90-487e-9a94-780475a94fb5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg"
Oct 30 20:50:07.989386 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:07.988492    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc"]
Oct 30 20:50:07.993852 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:07.993830    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc"
Oct 30 20:50:08.070680 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:08.070510    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whmpq\" (UniqueName: \"kubernetes.io/projected/2a7a1c0a-2a90-487e-9a94-780475a94fb5-kube-api-access-whmpq\") pod \"payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg\" (UID: \"2a7a1c0a-2a90-487e-9a94-780475a94fb5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg"
Oct 30 20:50:08.084362 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:08.084294    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc"]
Oct 30 20:50:08.089486 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:08.089416    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1bf6069-9c41-4f32-83c6-91561d7833b4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc\" (UID: \"f1bf6069-9c41-4f32-83c6-91561d7833b4\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc"
Oct 30 20:50:08.089584 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:08.089487    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn47m\" (UniqueName: \"kubernetes.io/projected/f1bf6069-9c41-4f32-83c6-91561d7833b4-kube-api-access-kn47m\") pod \"payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc\" (UID: \"f1bf6069-9c41-4f32-83c6-91561d7833b4\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc"
Oct 30 20:50:08.155575 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:08.155540    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg"
Oct 30 20:50:08.191238 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:08.191202    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1bf6069-9c41-4f32-83c6-91561d7833b4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc\" (UID: \"f1bf6069-9c41-4f32-83c6-91561d7833b4\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc"
Oct 30 20:50:08.191390 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:08.191272    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn47m\" (UniqueName: \"kubernetes.io/projected/f1bf6069-9c41-4f32-83c6-91561d7833b4-kube-api-access-kn47m\") pod \"payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc\" (UID: \"f1bf6069-9c41-4f32-83c6-91561d7833b4\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc"
Oct 30 20:50:08.191915 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:08.191894    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1bf6069-9c41-4f32-83c6-91561d7833b4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc\" (UID: \"f1bf6069-9c41-4f32-83c6-91561d7833b4\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc"
Oct 30 20:50:08.251276 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:08.251242    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn47m\" (UniqueName: \"kubernetes.io/projected/f1bf6069-9c41-4f32-83c6-91561d7833b4-kube-api-access-kn47m\") pod \"payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc\" (UID: \"f1bf6069-9c41-4f32-83c6-91561d7833b4\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc"
Oct 30 20:50:08.360706 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:08.360630    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc"
Oct 30 20:50:08.541038 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:08.541011    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg"]
Oct 30 20:50:08.542211 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:50:08.542188    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7a1c0a_2a90_487e_9a94_780475a94fb5.slice/crio-0dbb5d4658c394b7ce298d784e6bf85683add6780df0fda8aa61fe49f4427965 WatchSource:0}: Error finding container 0dbb5d4658c394b7ce298d784e6bf85683add6780df0fda8aa61fe49f4427965: Status 404 returned error can't find the container with id 0dbb5d4658c394b7ce298d784e6bf85683add6780df0fda8aa61fe49f4427965
Oct 30 20:50:08.738715 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:08.738689    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc"]
Oct 30 20:50:08.739106 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:50:08.739079    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1bf6069_9c41_4f32_83c6_91561d7833b4.slice/crio-dd35beb89d5454433ddf17a3516f2693b46c9e4fa5ecbaab5db234722145e96c WatchSource:0}: Error finding container dd35beb89d5454433ddf17a3516f2693b46c9e4fa5ecbaab5db234722145e96c: Status 404 returned error can't find the container with id dd35beb89d5454433ddf17a3516f2693b46c9e4fa5ecbaab5db234722145e96c
Oct 30 20:50:09.303588 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:09.303536    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc" event={"ID":"f1bf6069-9c41-4f32-83c6-91561d7833b4","Type":"ContainerStarted","Data":"c06fc3f444ae8103b5517f4e352dd6906bbbce380a45f72db00fbe88fa0d949d"}
Oct 30 20:50:09.304114 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:09.303617    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc"
Oct 30 20:50:09.304114 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:09.303642    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc" event={"ID":"f1bf6069-9c41-4f32-83c6-91561d7833b4","Type":"ContainerStarted","Data":"dd35beb89d5454433ddf17a3516f2693b46c9e4fa5ecbaab5db234722145e96c"}
Oct 30 20:50:09.306341 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:09.306299    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg" event={"ID":"2a7a1c0a-2a90-487e-9a94-780475a94fb5","Type":"ContainerStarted","Data":"dc9077c6115590b6b77e3374ab1f380edf4c6672984fd17effe8cdd2be8874e9"}
Oct 30 20:50:09.306467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:09.306394    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg" event={"ID":"2a7a1c0a-2a90-487e-9a94-780475a94fb5","Type":"ContainerStarted","Data":"0dbb5d4658c394b7ce298d784e6bf85683add6780df0fda8aa61fe49f4427965"}
Oct 30 20:50:09.306467 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:09.306424    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg"
Oct 30 20:50:09.418826 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:09.418761    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc" podStartSLOduration=2.418743477 podStartE2EDuration="2.418743477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:09.417443899 +0000 UTC m=+1829.096072516" watchObservedRunningTime="2026-10-30 20:50:09.418743477 +0000 UTC m=+1829.097372093"
Oct 30 20:50:09.542894 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:09.542833    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg" podStartSLOduration=2.5428107620000002 podStartE2EDuration="2.542810762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:09.540644263 +0000 UTC m=+1829.219272892" watchObservedRunningTime="2026-10-30 20:50:09.542810762 +0000 UTC m=+1829.221439381"
Oct 30 20:50:20.315655 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:20.315624    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc"
Oct 30 20:50:20.316167 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:20.315726    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg"
Oct 30 20:50:29.783846 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:29.783758    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh"]
Oct 30 20:50:29.789145 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:29.789120    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh"
Oct 30 20:50:29.841579 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:29.841552    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3f09ec\""
Oct 30 20:50:29.841918 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:29.841901    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3f09ec-dockercfg-2fr9r\""
Oct 30 20:50:29.894766 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:29.894728    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh"]
Oct 30 20:50:29.905805 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:29.905772    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdhfh\" (UniqueName: \"kubernetes.io/projected/84057aab-629f-4918-9d85-cd3e2d3a95b4-kube-api-access-sdhfh\") pod \"payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh\" (UID: \"84057aab-629f-4918-9d85-cd3e2d3a95b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh"
Oct 30 20:50:29.905975 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:29.905951    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84057aab-629f-4918-9d85-cd3e2d3a95b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh\" (UID: \"84057aab-629f-4918-9d85-cd3e2d3a95b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh"
Oct 30 20:50:30.010357 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:30.007995    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdhfh\" (UniqueName: \"kubernetes.io/projected/84057aab-629f-4918-9d85-cd3e2d3a95b4-kube-api-access-sdhfh\") pod \"payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh\" (UID: \"84057aab-629f-4918-9d85-cd3e2d3a95b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh"
Oct 30 20:50:30.010357 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:30.008140    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84057aab-629f-4918-9d85-cd3e2d3a95b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh\" (UID: \"84057aab-629f-4918-9d85-cd3e2d3a95b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh"
Oct 30 20:50:30.010357 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:30.009219    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84057aab-629f-4918-9d85-cd3e2d3a95b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh\" (UID: \"84057aab-629f-4918-9d85-cd3e2d3a95b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh"
Oct 30 20:50:30.105394 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:30.102288    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdhfh\" (UniqueName: \"kubernetes.io/projected/84057aab-629f-4918-9d85-cd3e2d3a95b4-kube-api-access-sdhfh\") pod \"payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh\" (UID: \"84057aab-629f-4918-9d85-cd3e2d3a95b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh"
Oct 30 20:50:30.105394 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:30.105260    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh"
Oct 30 20:50:30.552146 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:30.552113    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh"]
Oct 30 20:50:30.553403 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:50:30.553378    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84057aab_629f_4918_9d85_cd3e2d3a95b4.slice/crio-b1636e70c00d6d611330a36343b82859dab8ccaea7030ab2328989328ca0f708 WatchSource:0}: Error finding container b1636e70c00d6d611330a36343b82859dab8ccaea7030ab2328989328ca0f708: Status 404 returned error can't find the container with id b1636e70c00d6d611330a36343b82859dab8ccaea7030ab2328989328ca0f708
Oct 30 20:50:30.957647 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:50:30.957603    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:50:31.517897 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:31.517846    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh" event={"ID":"84057aab-629f-4918-9d85-cd3e2d3a95b4","Type":"ContainerStarted","Data":"b004aefefe6b5748bc835f26c1b7ee69d1487997deb22362f087ba748bbaebcf"}
Oct 30 20:50:31.518106 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:31.517917    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh"
Oct 30 20:50:31.518106 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:31.517941    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh" event={"ID":"84057aab-629f-4918-9d85-cd3e2d3a95b4","Type":"ContainerStarted","Data":"b1636e70c00d6d611330a36343b82859dab8ccaea7030ab2328989328ca0f708"}
Oct 30 20:50:31.640806 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:31.640745    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh" podStartSLOduration=2.640729022 podStartE2EDuration="2.640729022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:31.638235942 +0000 UTC m=+1851.316864559" watchObservedRunningTime="2026-10-30 20:50:31.640729022 +0000 UTC m=+1851.319357638"
Oct 30 20:50:40.294368 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.293886    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4"]
Oct 30 20:50:40.313783 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.313739    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4"
Oct 30 20:50:40.385906 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.385857    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pdfp\" (UniqueName: \"kubernetes.io/projected/67597528-00c9-4a1d-a5ed-02a170390a40-kube-api-access-7pdfp\") pod \"payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4\" (UID: \"67597528-00c9-4a1d-a5ed-02a170390a40\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4"
Oct 30 20:50:40.386355 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.386302    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67597528-00c9-4a1d-a5ed-02a170390a40-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4\" (UID: \"67597528-00c9-4a1d-a5ed-02a170390a40\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4"
Oct 30 20:50:40.398343 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.396653    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4"]
Oct 30 20:50:40.408737 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.408676    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-712099-dockercfg-www25\""
Oct 30 20:50:40.409675 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.409651    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-712099\""
Oct 30 20:50:40.476743 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.476703    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz"]
Oct 30 20:50:40.487804 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.487771    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67597528-00c9-4a1d-a5ed-02a170390a40-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4\" (UID: \"67597528-00c9-4a1d-a5ed-02a170390a40\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4"
Oct 30 20:50:40.488013 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.487838    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pdfp\" (UniqueName: \"kubernetes.io/projected/67597528-00c9-4a1d-a5ed-02a170390a40-kube-api-access-7pdfp\") pod \"payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4\" (UID: \"67597528-00c9-4a1d-a5ed-02a170390a40\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4"
Oct 30 20:50:40.496205 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.496179    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz"
Oct 30 20:50:40.539418 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.539374    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-712099\""
Oct 30 20:50:40.542865 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.542840    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz"]
Oct 30 20:50:40.549361 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.549272    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67597528-00c9-4a1d-a5ed-02a170390a40-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4\" (UID: \"67597528-00c9-4a1d-a5ed-02a170390a40\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4"
Oct 30 20:50:40.552143 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.552113    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pdfp\" (UniqueName: \"kubernetes.io/projected/67597528-00c9-4a1d-a5ed-02a170390a40-kube-api-access-7pdfp\") pod \"payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4\" (UID: \"67597528-00c9-4a1d-a5ed-02a170390a40\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4"
Oct 30 20:50:40.589558 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.589527    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkzdp\" (UniqueName: \"kubernetes.io/projected/4d739d8d-97fe-4c71-a9f4-d8d72ea853b6-kube-api-access-nkzdp\") pod \"payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz\" (UID: \"4d739d8d-97fe-4c71-a9f4-d8d72ea853b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz"
Oct 30 20:50:40.589737 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.589570    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d739d8d-97fe-4c71-a9f4-d8d72ea853b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz\" (UID: \"4d739d8d-97fe-4c71-a9f4-d8d72ea853b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz"
Oct 30 20:50:40.691472 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.691422    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkzdp\" (UniqueName: \"kubernetes.io/projected/4d739d8d-97fe-4c71-a9f4-d8d72ea853b6-kube-api-access-nkzdp\") pod \"payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz\" (UID: \"4d739d8d-97fe-4c71-a9f4-d8d72ea853b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz"
Oct 30 20:50:40.691472 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.691485    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d739d8d-97fe-4c71-a9f4-d8d72ea853b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz\" (UID: \"4d739d8d-97fe-4c71-a9f4-d8d72ea853b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz"
Oct 30 20:50:40.692729 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.692702    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d739d8d-97fe-4c71-a9f4-d8d72ea853b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz\" (UID: \"4d739d8d-97fe-4c71-a9f4-d8d72ea853b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz"
Oct 30 20:50:40.774148 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.774119    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkzdp\" (UniqueName: \"kubernetes.io/projected/4d739d8d-97fe-4c71-a9f4-d8d72ea853b6-kube-api-access-nkzdp\") pod \"payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz\" (UID: \"4d739d8d-97fe-4c71-a9f4-d8d72ea853b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz"
Oct 30 20:50:40.784384 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.784361    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-712099-dockercfg-www25\""
Oct 30 20:50:40.793732 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.793713    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4"
Oct 30 20:50:40.815165 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:40.815138    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz"
Oct 30 20:50:41.211498 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:41.211455    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4"]
Oct 30 20:50:41.234964 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:50:41.234930    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67597528_00c9_4a1d_a5ed_02a170390a40.slice/crio-f1357445968b4e8a84606da9c4a462726a529f15196ef3d04d2cb021749175fe WatchSource:0}: Error finding container f1357445968b4e8a84606da9c4a462726a529f15196ef3d04d2cb021749175fe: Status 404 returned error can't find the container with id f1357445968b4e8a84606da9c4a462726a529f15196ef3d04d2cb021749175fe
Oct 30 20:50:41.253533 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:41.253506    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz"]
Oct 30 20:50:41.254427 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:50:41.254400    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d739d8d_97fe_4c71_a9f4_d8d72ea853b6.slice/crio-e138c992cbecc03a6d989dea9d6482aed372ffc5d52125bbcffee3bf0987fa9c WatchSource:0}: Error finding container e138c992cbecc03a6d989dea9d6482aed372ffc5d52125bbcffee3bf0987fa9c: Status 404 returned error can't find the container with id e138c992cbecc03a6d989dea9d6482aed372ffc5d52125bbcffee3bf0987fa9c
Oct 30 20:50:41.583649 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:41.583552    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4" event={"ID":"67597528-00c9-4a1d-a5ed-02a170390a40","Type":"ContainerStarted","Data":"cf883912f9e50d568fb5d02c424d9e3d0c25b306a2a80c9650303adc4dc91b31"}
Oct 30 20:50:41.583649 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:41.583622    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4"
Oct 30 20:50:41.584144 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:41.583652    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4" event={"ID":"67597528-00c9-4a1d-a5ed-02a170390a40","Type":"ContainerStarted","Data":"f1357445968b4e8a84606da9c4a462726a529f15196ef3d04d2cb021749175fe"}
Oct 30 20:50:41.585608 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:41.585577    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz" event={"ID":"4d739d8d-97fe-4c71-a9f4-d8d72ea853b6","Type":"ContainerStarted","Data":"f51c66cb102a3d8b6f4b5a432f4b74dbdb2e3d1f00ae3ff0c28b8b0815034ef8"}
Oct 30 20:50:41.585743 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:41.585622    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz" event={"ID":"4d739d8d-97fe-4c71-a9f4-d8d72ea853b6","Type":"ContainerStarted","Data":"e138c992cbecc03a6d989dea9d6482aed372ffc5d52125bbcffee3bf0987fa9c"}
Oct 30 20:50:41.585743 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:41.585650    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz"
Oct 30 20:50:41.701192 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:41.701125    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4" podStartSLOduration=1.7011062030000001 podStartE2EDuration="1.701106203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:41.698274959 +0000 UTC m=+1861.376903576" watchObservedRunningTime="2026-10-30 20:50:41.701106203 +0000 UTC m=+1861.379734820"
Oct 30 20:50:42.527892 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:42.527859    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh"
Oct 30 20:50:42.648137 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:42.648084    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz" podStartSLOduration=2.6480690129999997 podStartE2EDuration="2.648069013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:41.854437523 +0000 UTC m=+1861.533066142" watchObservedRunningTime="2026-10-30 20:50:42.648069013 +0000 UTC m=+1862.326697628"
Oct 30 20:50:52.594955 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:52.594922    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz"
Oct 30 20:50:52.595503 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:52.595092    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4"
Oct 30 20:50:54.321113 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:54.320487    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7g5ll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:50:54.321113 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:54.320627    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll" podUID="1ac0a5d8-9d3d-4a31-ac67-16c610d242ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:50:56.416553 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:56.416516    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"]
Oct 30 20:50:56.421185 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:56.421160    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"
Oct 30 20:50:56.442394 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:56.442371    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec075762-7e92-471c-bc35-c456ab936ad0-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx\" (UID: \"ec075762-7e92-471c-bc35-c456ab936ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"
Oct 30 20:50:56.442518 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:56.442402    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqxdq\" (UniqueName: \"kubernetes.io/projected/ec075762-7e92-471c-bc35-c456ab936ad0-kube-api-access-dqxdq\") pod \"maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx\" (UID: \"ec075762-7e92-471c-bc35-c456ab936ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"
Oct 30 20:50:56.442592 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:56.442570    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec075762-7e92-471c-bc35-c456ab936ad0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx\" (UID: \"ec075762-7e92-471c-bc35-c456ab936ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"
Oct 30 20:50:56.442629 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:56.442608    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec075762-7e92-471c-bc35-c456ab936ad0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx\" (UID: \"ec075762-7e92-471c-bc35-c456ab936ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"
Oct 30 20:50:56.473674 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:56.473650    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-f8ac60-serving-cert\""
Oct 30 20:50:56.527781 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:56.527746    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"]
Oct 30 20:50:56.544032 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:56.544002    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec075762-7e92-471c-bc35-c456ab936ad0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx\" (UID: \"ec075762-7e92-471c-bc35-c456ab936ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"
Oct 30 20:50:56.544202 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:56.544047    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec075762-7e92-471c-bc35-c456ab936ad0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx\" (UID: \"ec075762-7e92-471c-bc35-c456ab936ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"
Oct 30 20:50:56.544202 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:50:56.544149    4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-serving-cert: secret "maas-api-e2e-shared-b-w2-f8ac60-serving-cert" not found
Oct 30 20:50:56.544274 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:50:56.544216    4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec075762-7e92-471c-bc35-c456ab936ad0-maas-api-tls podName:ec075762-7e92-471c-bc35-c456ab936ad0 nodeName:}" failed. No retries permitted until 2026-10-30 20:50:57.044196741 +0000 UTC m=+1876.722825334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec075762-7e92-471c-bc35-c456ab936ad0-maas-api-tls") pod "maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx" (UID: "ec075762-7e92-471c-bc35-c456ab936ad0") : secret "maas-api-e2e-shared-b-w2-f8ac60-serving-cert" not found
Oct 30 20:50:56.544274 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:56.544237    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec075762-7e92-471c-bc35-c456ab936ad0-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx\" (UID: \"ec075762-7e92-471c-bc35-c456ab936ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"
Oct 30 20:50:56.544274 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:56.544258    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqxdq\" (UniqueName: \"kubernetes.io/projected/ec075762-7e92-471c-bc35-c456ab936ad0-kube-api-access-dqxdq\") pod \"maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx\" (UID: \"ec075762-7e92-471c-bc35-c456ab936ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"
Oct 30 20:50:56.544875 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:56.544836    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec075762-7e92-471c-bc35-c456ab936ad0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx\" (UID: \"ec075762-7e92-471c-bc35-c456ab936ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"
Oct 30 20:50:56.546807 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:56.546780    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec075762-7e92-471c-bc35-c456ab936ad0-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx\" (UID: \"ec075762-7e92-471c-bc35-c456ab936ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"
Oct 30 20:50:56.604517 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:56.604487    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqxdq\" (UniqueName: \"kubernetes.io/projected/ec075762-7e92-471c-bc35-c456ab936ad0-kube-api-access-dqxdq\") pod \"maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx\" (UID: \"ec075762-7e92-471c-bc35-c456ab936ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"
Oct 30 20:50:57.053850 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:57.053814    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec075762-7e92-471c-bc35-c456ab936ad0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx\" (UID: \"ec075762-7e92-471c-bc35-c456ab936ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"
Oct 30 20:50:57.056562 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:57.056536    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec075762-7e92-471c-bc35-c456ab936ad0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx\" (UID: \"ec075762-7e92-471c-bc35-c456ab936ad0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"
Oct 30 20:50:57.335860 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:57.335772    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"
Oct 30 20:50:57.717008 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:57.716975    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"]
Oct 30 20:50:57.717971 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:50:57.717946    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec075762_7e92_471c_bc35_c456ab936ad0.slice/crio-22ad219649555dfb4a9a82a1eaa44094452ebcbf1b0d1974f7fa73aee6cf4043 WatchSource:0}: Error finding container 22ad219649555dfb4a9a82a1eaa44094452ebcbf1b0d1974f7fa73aee6cf4043: Status 404 returned error can't find the container with id 22ad219649555dfb4a9a82a1eaa44094452ebcbf1b0d1974f7fa73aee6cf4043
Oct 30 20:50:57.739879 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:57.739850    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx" event={"ID":"ec075762-7e92-471c-bc35-c456ab936ad0","Type":"ContainerStarted","Data":"22ad219649555dfb4a9a82a1eaa44094452ebcbf1b0d1974f7fa73aee6cf4043"}
Oct 30 20:50:58.749384 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:58.749351    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx" event={"ID":"ec075762-7e92-471c-bc35-c456ab936ad0","Type":"ContainerStarted","Data":"1f771057e49da49e2ccb9499f4089dbf7ffcb70fe9b050b74beaa33045209a3a"}
Oct 30 20:50:58.749878 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:58.749556    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"
Oct 30 20:50:58.893980 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:50:58.893927    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx" podStartSLOduration=2.06861584 podStartE2EDuration="2.893912993s" totalImagesPullingTime="825.297153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:58.891830576 +0000 UTC m=+1878.570459229" watchObservedRunningTime="2026-10-30 20:50:58.893912993 +0000 UTC m=+1878.572541610"
Oct 30 20:51:00.954478 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:51:00.954442    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:51:04.762129 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:04.762096    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"
Oct 30 20:51:09.387227 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:09.387186    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7269-b4eff6"]
Oct 30 20:51:09.391505 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:09.391481    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-b4eff6"
Oct 30 20:51:09.435762 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:09.435723    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55kdl\" (UniqueName: \"kubernetes.io/projected/a56e89e2-112b-4937-8da0-70e04bedd0c4-kube-api-access-55kdl\") pod \"test-curl-7269-b4eff6\" (UID: \"a56e89e2-112b-4937-8da0-70e04bedd0c4\") " pod="opendatahub/test-curl-7269-b4eff6"
Oct 30 20:51:09.447930 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:09.447907    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7269-b4eff6"]
Oct 30 20:51:09.493293 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:09.493263    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vjnsm\""
Oct 30 20:51:09.536595 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:09.536566    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55kdl\" (UniqueName: \"kubernetes.io/projected/a56e89e2-112b-4937-8da0-70e04bedd0c4-kube-api-access-55kdl\") pod \"test-curl-7269-b4eff6\" (UID: \"a56e89e2-112b-4937-8da0-70e04bedd0c4\") " pod="opendatahub/test-curl-7269-b4eff6"
Oct 30 20:51:09.593285 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:09.593246    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55kdl\" (UniqueName: \"kubernetes.io/projected/a56e89e2-112b-4937-8da0-70e04bedd0c4-kube-api-access-55kdl\") pod \"test-curl-7269-b4eff6\" (UID: \"a56e89e2-112b-4937-8da0-70e04bedd0c4\") " pod="opendatahub/test-curl-7269-b4eff6"
Oct 30 20:51:09.704212 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:09.704141    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-b4eff6"
Oct 30 20:51:10.078836 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:10.078804    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7269-b4eff6"]
Oct 30 20:51:10.080065 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:51:10.080039    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda56e89e2_112b_4937_8da0_70e04bedd0c4.slice/crio-141fcdc1d569a60aa598d215c7c1c417921e7f08ae6db4264581ddb44e3d7199 WatchSource:0}: Error finding container 141fcdc1d569a60aa598d215c7c1c417921e7f08ae6db4264581ddb44e3d7199: Status 404 returned error can't find the container with id 141fcdc1d569a60aa598d215c7c1c417921e7f08ae6db4264581ddb44e3d7199
Oct 30 20:51:10.823915 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:10.823877    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7269-b4eff6" event={"ID":"a56e89e2-112b-4937-8da0-70e04bedd0c4","Type":"ContainerStarted","Data":"b43c6af490bbc9a6817e74c13b76ae736bf73649874dfb9926aa076ac17b9fe6"}
Oct 30 20:51:10.824342 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:10.823930    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7269-b4eff6" event={"ID":"a56e89e2-112b-4937-8da0-70e04bedd0c4","Type":"ContainerStarted","Data":"141fcdc1d569a60aa598d215c7c1c417921e7f08ae6db4264581ddb44e3d7199"}
Oct 30 20:51:12.497429 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:12.497368    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7269-b4eff6" podStartSLOduration=3.49735172 podStartE2EDuration="3.49735172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:10.955121702 +0000 UTC m=+1890.633750317" watchObservedRunningTime="2026-10-30 20:51:12.49735172 +0000 UTC m=+1892.175980333"
Oct 30 20:51:12.500524 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:12.500503    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7269-b4eff6"]
Oct 30 20:51:12.506083 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:12.506058    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7269-b4eff6"]
Oct 30 20:51:12.835157 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:12.835054    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7269-b4eff6" podUID="a56e89e2-112b-4937-8da0-70e04bedd0c4" containerName="test-curl-7269-b4eff6" containerID="cri-o://b43c6af490bbc9a6817e74c13b76ae736bf73649874dfb9926aa076ac17b9fe6" gracePeriod=2
Oct 30 20:51:12.886103 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:51:12.886064    4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-b4eff6\" is forbidden: User \"system:node:ip-10-0-138-34.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-34.us-west-1.compute.internal' and this object" podUID="a56e89e2-112b-4937-8da0-70e04bedd0c4" pod="opendatahub/test-curl-7269-b4eff6"
Oct 30 20:51:15.088178 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:15.088153    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-b4eff6"
Oct 30 20:51:15.210667 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:15.210636    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a56e89e2-112b-4937-8da0-70e04bedd0c4-kube-api-access-55kdl\" (UniqueName: \"kubernetes.io/projected/a56e89e2-112b-4937-8da0-70e04bedd0c4-kube-api-access-55kdl\") pod \"a56e89e2-112b-4937-8da0-70e04bedd0c4\" (UID: \"a56e89e2-112b-4937-8da0-70e04bedd0c4\") "
Oct 30 20:51:15.212955 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:15.212925    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a56e89e2-112b-4937-8da0-70e04bedd0c4-kube-api-access-55kdl" pod "a56e89e2-112b-4937-8da0-70e04bedd0c4" (UID: "a56e89e2-112b-4937-8da0-70e04bedd0c4"). InnerVolumeSpecName "kube-api-access-55kdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:51:15.312409 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:15.312375    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55kdl\" (UniqueName: \"kubernetes.io/projected/a56e89e2-112b-4937-8da0-70e04bedd0c4-kube-api-access-55kdl\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:51:15.855402 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:15.855369    4471 generic.go:356] "Generic (PLEG): container finished" podID="a56e89e2-112b-4937-8da0-70e04bedd0c4" containerID="b43c6af490bbc9a6817e74c13b76ae736bf73649874dfb9926aa076ac17b9fe6" exitCode=137
Oct 30 20:51:15.855610 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:15.855424    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7269-b4eff6"
Oct 30 20:51:15.855610 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:15.855454    4471 scope.go:122] "RemoveContainer" containerID="b43c6af490bbc9a6817e74c13b76ae736bf73649874dfb9926aa076ac17b9fe6"
Oct 30 20:51:15.865719 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:15.865696    4471 scope.go:122] "RemoveContainer" containerID="b43c6af490bbc9a6817e74c13b76ae736bf73649874dfb9926aa076ac17b9fe6"
Oct 30 20:51:15.865976 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:51:15.865957    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b43c6af490bbc9a6817e74c13b76ae736bf73649874dfb9926aa076ac17b9fe6\": container with ID starting with b43c6af490bbc9a6817e74c13b76ae736bf73649874dfb9926aa076ac17b9fe6 not found: ID does not exist" containerID="b43c6af490bbc9a6817e74c13b76ae736bf73649874dfb9926aa076ac17b9fe6"
Oct 30 20:51:15.866049 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:15.865984    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b43c6af490bbc9a6817e74c13b76ae736bf73649874dfb9926aa076ac17b9fe6"} err="failed to get container status \"b43c6af490bbc9a6817e74c13b76ae736bf73649874dfb9926aa076ac17b9fe6\": rpc error: code = NotFound desc = could not find container \"b43c6af490bbc9a6817e74c13b76ae736bf73649874dfb9926aa076ac17b9fe6\": container with ID starting with b43c6af490bbc9a6817e74c13b76ae736bf73649874dfb9926aa076ac17b9fe6 not found: ID does not exist"
Oct 30 20:51:15.915307 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:51:15.915278    4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7269-b4eff6\" is forbidden: User \"system:node:ip-10-0-138-34.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-34.us-west-1.compute.internal' and this object" podUID="a56e89e2-112b-4937-8da0-70e04bedd0c4" pod="opendatahub/test-curl-7269-b4eff6"
Oct 30 20:51:16.459669 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:16.459602    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a56e89e2-112b-4937-8da0-70e04bedd0c4" path="/var/lib/kubelet/pods/a56e89e2-112b-4937-8da0-70e04bedd0c4/volumes"
Oct 30 20:51:30.957499 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:51:30.957455    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:51:50.618476 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:50.618430    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"]
Oct 30 20:51:50.619069 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:50.619052    4471 cpu_manager.go:414] "containerMap: removing container" podUID="a56e89e2-112b-4937-8da0-70e04bedd0c4" containerName="test-curl-7269-b4eff6"
Oct 30 20:51:50.619069 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:50.619069    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a56e89e2-112b-4937-8da0-70e04bedd0c4" containerName="test-curl-7269-b4eff6"
Oct 30 20:51:50.619216 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:50.619163    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="a56e89e2-112b-4937-8da0-70e04bedd0c4" containerName="test-curl-7269-b4eff6"
Oct 30 20:51:50.623755 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:50.623734    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"
Oct 30 20:51:50.675500 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:50.675468    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3f09ec-serving-cert\""
Oct 30 20:51:50.720487 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:50.720450    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2199190-0e92-4238-8011-84a7d273a53e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv\" (UID: \"a2199190-0e92-4238-8011-84a7d273a53e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"
Oct 30 20:51:50.720634 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:50.720515    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2199190-0e92-4238-8011-84a7d273a53e-maas-api-tls\") pod \"maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv\" (UID: \"a2199190-0e92-4238-8011-84a7d273a53e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"
Oct 30 20:51:50.720744 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:50.720715    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2199190-0e92-4238-8011-84a7d273a53e-metrics-certs\") pod \"maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv\" (UID: \"a2199190-0e92-4238-8011-84a7d273a53e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"
Oct 30 20:51:50.720896 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:50.720878    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqxsm\" (UniqueName: \"kubernetes.io/projected/a2199190-0e92-4238-8011-84a7d273a53e-kube-api-access-tqxsm\") pod \"maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv\" (UID: \"a2199190-0e92-4238-8011-84a7d273a53e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"
Oct 30 20:51:50.728192 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:50.728164    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"]
Oct 30 20:51:50.822226 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:50.822184    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqxsm\" (UniqueName: \"kubernetes.io/projected/a2199190-0e92-4238-8011-84a7d273a53e-kube-api-access-tqxsm\") pod \"maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv\" (UID: \"a2199190-0e92-4238-8011-84a7d273a53e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"
Oct 30 20:51:50.822446 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:50.822278    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2199190-0e92-4238-8011-84a7d273a53e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv\" (UID: \"a2199190-0e92-4238-8011-84a7d273a53e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"
Oct 30 20:51:50.822446 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:50.822309    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2199190-0e92-4238-8011-84a7d273a53e-maas-api-tls\") pod \"maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv\" (UID: \"a2199190-0e92-4238-8011-84a7d273a53e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"
Oct 30 20:51:50.822446 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:50.822425    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2199190-0e92-4238-8011-84a7d273a53e-metrics-certs\") pod \"maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv\" (UID: \"a2199190-0e92-4238-8011-84a7d273a53e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"
Oct 30 20:51:50.822613 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:51:50.822498    4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-serving-cert: secret "maas-api-e2e-inf-a-3f09ec-serving-cert" not found
Oct 30 20:51:50.822613 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:51:50.822572    4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2199190-0e92-4238-8011-84a7d273a53e-maas-api-tls podName:a2199190-0e92-4238-8011-84a7d273a53e nodeName:}" failed. No retries permitted until 2026-10-30 20:51:51.322557755 +0000 UTC m=+1931.001186381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a2199190-0e92-4238-8011-84a7d273a53e-maas-api-tls") pod "maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv" (UID: "a2199190-0e92-4238-8011-84a7d273a53e") : secret "maas-api-e2e-inf-a-3f09ec-serving-cert" not found
Oct 30 20:51:50.823066 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:50.823048    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2199190-0e92-4238-8011-84a7d273a53e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv\" (UID: \"a2199190-0e92-4238-8011-84a7d273a53e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"
Oct 30 20:51:50.824985 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:50.824962    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2199190-0e92-4238-8011-84a7d273a53e-metrics-certs\") pod \"maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv\" (UID: \"a2199190-0e92-4238-8011-84a7d273a53e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"
Oct 30 20:51:50.891146 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:50.891073    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqxsm\" (UniqueName: \"kubernetes.io/projected/a2199190-0e92-4238-8011-84a7d273a53e-kube-api-access-tqxsm\") pod \"maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv\" (UID: \"a2199190-0e92-4238-8011-84a7d273a53e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"
Oct 30 20:51:51.337677 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:51.337583    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2199190-0e92-4238-8011-84a7d273a53e-maas-api-tls\") pod \"maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv\" (UID: \"a2199190-0e92-4238-8011-84a7d273a53e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"
Oct 30 20:51:51.340257 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:51.340229    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2199190-0e92-4238-8011-84a7d273a53e-maas-api-tls\") pod \"maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv\" (UID: \"a2199190-0e92-4238-8011-84a7d273a53e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"
Oct 30 20:51:51.537886 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:51.537855    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"
Oct 30 20:51:51.914305 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:51.914272    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"]
Oct 30 20:51:51.915737 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:51:51.915714    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2199190_0e92_4238_8011_84a7d273a53e.slice/crio-852238d92c22cd906c14e72ccbb1dbbfeecd2d9ed9fb324f52e40edeb11d3994 WatchSource:0}: Error finding container 852238d92c22cd906c14e72ccbb1dbbfeecd2d9ed9fb324f52e40edeb11d3994: Status 404 returned error can't find the container with id 852238d92c22cd906c14e72ccbb1dbbfeecd2d9ed9fb324f52e40edeb11d3994
Oct 30 20:51:51.917340 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:51.917299    4471 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:51:52.097592 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:52.097520    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv" event={"ID":"a2199190-0e92-4238-8011-84a7d273a53e","Type":"ContainerStarted","Data":"852238d92c22cd906c14e72ccbb1dbbfeecd2d9ed9fb324f52e40edeb11d3994"}
Oct 30 20:51:53.108085 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:53.107991    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv" event={"ID":"a2199190-0e92-4238-8011-84a7d273a53e","Type":"ContainerStarted","Data":"3f019ea9d03df5a06b88171744df8f4fd98d0849d7fc7b166732e9d1e28c2123"}
Oct 30 20:51:53.108580 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:53.108198    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"
Oct 30 20:51:53.230534 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:53.230479    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv" podStartSLOduration=2.3208725230000002 podStartE2EDuration="3.230462802s" totalImagesPullingTime="909.590279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:53.226548503 +0000 UTC m=+1932.905177120" watchObservedRunningTime="2026-10-30 20:51:53.230462802 +0000 UTC m=+1932.909091418"
Oct 30 20:51:59.122931 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:51:59.122900    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"
Oct 30 20:52:00.955133 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:52:00.955094    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:52:05.801867 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:05.801834    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"]
Oct 30 20:52:05.806043 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:05.806022    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"
Oct 30 20:52:05.857837 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:05.857807    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-712099-serving-cert\""
Oct 30 20:52:05.910217 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:05.910181    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"]
Oct 30 20:52:05.925996 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:05.925971    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c185fa5-18ff-4caf-b853-dee1d4b0432f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8\" (UID: \"4c185fa5-18ff-4caf-b853-dee1d4b0432f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"
Oct 30 20:52:05.926146 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:05.926109    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c185fa5-18ff-4caf-b853-dee1d4b0432f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8\" (UID: \"4c185fa5-18ff-4caf-b853-dee1d4b0432f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"
Oct 30 20:52:05.926146 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:05.926139    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgk4c\" (UniqueName: \"kubernetes.io/projected/4c185fa5-18ff-4caf-b853-dee1d4b0432f-kube-api-access-zgk4c\") pod \"maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8\" (UID: \"4c185fa5-18ff-4caf-b853-dee1d4b0432f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"
Oct 30 20:52:05.926256 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:05.926236    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c185fa5-18ff-4caf-b853-dee1d4b0432f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8\" (UID: \"4c185fa5-18ff-4caf-b853-dee1d4b0432f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"
Oct 30 20:52:06.027445 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:06.027415    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c185fa5-18ff-4caf-b853-dee1d4b0432f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8\" (UID: \"4c185fa5-18ff-4caf-b853-dee1d4b0432f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"
Oct 30 20:52:06.027445 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:06.027450    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgk4c\" (UniqueName: \"kubernetes.io/projected/4c185fa5-18ff-4caf-b853-dee1d4b0432f-kube-api-access-zgk4c\") pod \"maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8\" (UID: \"4c185fa5-18ff-4caf-b853-dee1d4b0432f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"
Oct 30 20:52:06.027701 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:06.027481    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c185fa5-18ff-4caf-b853-dee1d4b0432f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8\" (UID: \"4c185fa5-18ff-4caf-b853-dee1d4b0432f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"
Oct 30 20:52:06.027701 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:06.027600    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c185fa5-18ff-4caf-b853-dee1d4b0432f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8\" (UID: \"4c185fa5-18ff-4caf-b853-dee1d4b0432f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"
Oct 30 20:52:06.027803 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:52:06.027734    4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-serving-cert: secret "maas-api-e2e-shared-b-w3-712099-serving-cert" not found
Oct 30 20:52:06.027854 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:52:06.027806    4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c185fa5-18ff-4caf-b853-dee1d4b0432f-maas-api-tls podName:4c185fa5-18ff-4caf-b853-dee1d4b0432f nodeName:}" failed. No retries permitted until 2026-10-30 20:52:06.527788737 +0000 UTC m=+1946.206417342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4c185fa5-18ff-4caf-b853-dee1d4b0432f-maas-api-tls") pod "maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8" (UID: "4c185fa5-18ff-4caf-b853-dee1d4b0432f") : secret "maas-api-e2e-shared-b-w3-712099-serving-cert" not found
Oct 30 20:52:06.028247 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:06.028225    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c185fa5-18ff-4caf-b853-dee1d4b0432f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8\" (UID: \"4c185fa5-18ff-4caf-b853-dee1d4b0432f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"
Oct 30 20:52:06.030131 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:06.030109    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c185fa5-18ff-4caf-b853-dee1d4b0432f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8\" (UID: \"4c185fa5-18ff-4caf-b853-dee1d4b0432f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"
Oct 30 20:52:06.094861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:06.094648    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgk4c\" (UniqueName: \"kubernetes.io/projected/4c185fa5-18ff-4caf-b853-dee1d4b0432f-kube-api-access-zgk4c\") pod \"maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8\" (UID: \"4c185fa5-18ff-4caf-b853-dee1d4b0432f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"
Oct 30 20:52:06.534925 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:06.534890    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c185fa5-18ff-4caf-b853-dee1d4b0432f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8\" (UID: \"4c185fa5-18ff-4caf-b853-dee1d4b0432f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"
Oct 30 20:52:06.537665 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:06.537635    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c185fa5-18ff-4caf-b853-dee1d4b0432f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8\" (UID: \"4c185fa5-18ff-4caf-b853-dee1d4b0432f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"
Oct 30 20:52:06.718871 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:06.718839    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"
Oct 30 20:52:07.096573 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:07.096540    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"]
Oct 30 20:52:07.097882 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:52:07.097849    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c185fa5_18ff_4caf_b853_dee1d4b0432f.slice/crio-c5f0822008d80059e56a4a9908fb52c71632f41eea6bc1e4813a3848e7476210 WatchSource:0}: Error finding container c5f0822008d80059e56a4a9908fb52c71632f41eea6bc1e4813a3848e7476210: Status 404 returned error can't find the container with id c5f0822008d80059e56a4a9908fb52c71632f41eea6bc1e4813a3848e7476210
Oct 30 20:52:07.311227 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:07.311185    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8" event={"ID":"4c185fa5-18ff-4caf-b853-dee1d4b0432f","Type":"ContainerStarted","Data":"c5f0822008d80059e56a4a9908fb52c71632f41eea6bc1e4813a3848e7476210"}
Oct 30 20:52:08.319063 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:08.318971    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8" event={"ID":"4c185fa5-18ff-4caf-b853-dee1d4b0432f","Type":"ContainerStarted","Data":"14479cbd6454ed9359d134bcb08dd938abdebf6003de65e2a8d4658d99a29896"}
Oct 30 20:52:08.319477 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:08.319112    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"
Oct 30 20:52:08.437972 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:08.437922    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8" podStartSLOduration=2.540679182 podStartE2EDuration="3.437903705s" totalImagesPullingTime="897.224523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:08.434653601 +0000 UTC m=+1948.113282232" watchObservedRunningTime="2026-10-30 20:52:08.437903705 +0000 UTC m=+1948.116532323"
Oct 30 20:52:14.330644 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:14.330617    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"
Oct 30 20:52:27.086363 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:27.086331    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"]
Oct 30 20:52:27.086833 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:27.086557    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp" podUID="7b0db74e-64b2-4bb2-83da-ee1669156580" containerName="istio-proxy" containerID="cri-o://b392525196987dcd71139ce2bfca3ce121ce3c205f3aedc992387fa888915756" gracePeriod=30
Oct 30 20:52:27.135656 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:27.135626    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"]
Oct 30 20:52:27.135920 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:27.135897    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw" podUID="165b506b-c809-43eb-a183-686869cf35db" containerName="istio-proxy" containerID="cri-o://402de59011fb87953260c8e332542a0e94fff69cb91b94ecb368dde240f1ae31" gracePeriod=30
Oct 30 20:52:27.138549 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:27.138521    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"]
Oct 30 20:52:27.138756 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:27.138732    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx" podUID="19dfa0fc-a01f-495b-8d9f-e6e976bef433" containerName="istio-proxy" containerID="cri-o://c1896b4b544a8c43b11e19ce75d2cc44a13f9b42de61a5ede3370a5fa29a390e" gracePeriod=30
Oct 30 20:52:30.317217 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:30.317185    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"]
Oct 30 20:52:30.317663 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:30.317429    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7" podUID="d03eaf5f-735f-43cf-aa21-45901e9e6c04" containerName="istio-proxy" containerID="cri-o://a9ba5e64b7f49212415bc458bd5f924389f7ad6b78f6b828640773f77becd359" gracePeriod=30
Oct 30 20:52:30.401564 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:30.401524    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"]
Oct 30 20:52:30.401927 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:30.401896    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c" podUID="3613a886-6b08-4e25-807a-fbbe5c080775" containerName="istio-proxy" containerID="cri-o://901cb1872378aa2a25d3b86cff3a747984eece3970a3ce8fce5c2af913b6395a" gracePeriod=30
Oct 30 20:52:30.425050 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:30.425020    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"]
Oct 30 20:52:30.425380 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:30.425305    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr" podUID="03e1b048-5b3f-43fd-b321-897f8d6e6e53" containerName="istio-proxy" containerID="cri-o://cb0cfeec4ea49952fc4fa219e6e1137d91271b77ce04146f57e33e184851e66f" gracePeriod=30
Oct 30 20:52:30.853585 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:30.853544    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:52:30.853754 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:30.853624    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx" podUID="19dfa0fc-a01f-495b-8d9f-e6e976bef433" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:52:31.003932 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:52:31.003898    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:52:31.195255 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:31.195226    4471 patch_prober.go:29] interesting pod/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:52:31.195432 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:31.195288    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c" podUID="3613a886-6b08-4e25-807a-fbbe5c080775" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:52:32.007656 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.007623    4471 patch_prober.go:29] interesting pod/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:52:32.008062 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.007685    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp" podUID="7b0db74e-64b2-4bb2-83da-ee1669156580" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:52:32.428213 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.428181    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:52:32.470770 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.470743    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:52:32.473596 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.473568    4471 generic.go:356] "Generic (PLEG): container finished" podID="7b0db74e-64b2-4bb2-83da-ee1669156580" containerID="b392525196987dcd71139ce2bfca3ce121ce3c205f3aedc992387fa888915756" exitCode=0
Oct 30 20:52:32.473718 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.473616    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"
Oct 30 20:52:32.473718 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.473645    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp" event={"ID":"7b0db74e-64b2-4bb2-83da-ee1669156580","Type":"ContainerDied","Data":"b392525196987dcd71139ce2bfca3ce121ce3c205f3aedc992387fa888915756"}
Oct 30 20:52:32.473718 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.473697    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp" event={"ID":"7b0db74e-64b2-4bb2-83da-ee1669156580","Type":"ContainerDied","Data":"afc317edb761265e22881a0a4ddad25bcf0a761d495e73b2283c5fcfee31dc16"}
Oct 30 20:52:32.473875 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.473727    4471 scope.go:122] "RemoveContainer" containerID="b392525196987dcd71139ce2bfca3ce121ce3c205f3aedc992387fa888915756"
Oct 30 20:52:32.475190 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.475172    4471 generic.go:356] "Generic (PLEG): container finished" podID="19dfa0fc-a01f-495b-8d9f-e6e976bef433" containerID="c1896b4b544a8c43b11e19ce75d2cc44a13f9b42de61a5ede3370a5fa29a390e" exitCode=0
Oct 30 20:52:32.475295 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.475276    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx" event={"ID":"19dfa0fc-a01f-495b-8d9f-e6e976bef433","Type":"ContainerDied","Data":"c1896b4b544a8c43b11e19ce75d2cc44a13f9b42de61a5ede3370a5fa29a390e"}
Oct 30 20:52:32.475383 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.475339    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx" event={"ID":"19dfa0fc-a01f-495b-8d9f-e6e976bef433","Type":"ContainerDied","Data":"ccaf3baa9d872f52b73e18a8123aa41ef9d9b13ed860b75047ffb1f2e8728b30"}
Oct 30 20:52:32.475383 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.475345    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"
Oct 30 20:52:32.477166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.477142    4471 generic.go:356] "Generic (PLEG): container finished" podID="165b506b-c809-43eb-a183-686869cf35db" containerID="402de59011fb87953260c8e332542a0e94fff69cb91b94ecb368dde240f1ae31" exitCode=0
Oct 30 20:52:32.477267 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.477211    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw" event={"ID":"165b506b-c809-43eb-a183-686869cf35db","Type":"ContainerDied","Data":"402de59011fb87953260c8e332542a0e94fff69cb91b94ecb368dde240f1ae31"}
Oct 30 20:52:32.484349 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.484317    4471 scope.go:122] "RemoveContainer" containerID="b392525196987dcd71139ce2bfca3ce121ce3c205f3aedc992387fa888915756"
Oct 30 20:52:32.484649 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:52:32.484621    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b392525196987dcd71139ce2bfca3ce121ce3c205f3aedc992387fa888915756\": container with ID starting with b392525196987dcd71139ce2bfca3ce121ce3c205f3aedc992387fa888915756 not found: ID does not exist" containerID="b392525196987dcd71139ce2bfca3ce121ce3c205f3aedc992387fa888915756"
Oct 30 20:52:32.484753 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.484653    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b392525196987dcd71139ce2bfca3ce121ce3c205f3aedc992387fa888915756"} err="failed to get container status \"b392525196987dcd71139ce2bfca3ce121ce3c205f3aedc992387fa888915756\": rpc error: code = NotFound desc = could not find container \"b392525196987dcd71139ce2bfca3ce121ce3c205f3aedc992387fa888915756\": container with ID starting with b392525196987dcd71139ce2bfca3ce121ce3c205f3aedc992387fa888915756 not found: ID does not exist"
Oct 30 20:52:32.484753 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.484677    4471 scope.go:122] "RemoveContainer" containerID="c1896b4b544a8c43b11e19ce75d2cc44a13f9b42de61a5ede3370a5fa29a390e"
Oct 30 20:52:32.494129 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.494107    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:52:32.495644 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.495626    4471 scope.go:122] "RemoveContainer" containerID="c1896b4b544a8c43b11e19ce75d2cc44a13f9b42de61a5ede3370a5fa29a390e"
Oct 30 20:52:32.495926 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:52:32.495905    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1896b4b544a8c43b11e19ce75d2cc44a13f9b42de61a5ede3370a5fa29a390e\": container with ID starting with c1896b4b544a8c43b11e19ce75d2cc44a13f9b42de61a5ede3370a5fa29a390e not found: ID does not exist" containerID="c1896b4b544a8c43b11e19ce75d2cc44a13f9b42de61a5ede3370a5fa29a390e"
Oct 30 20:52:32.495980 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.495935    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1896b4b544a8c43b11e19ce75d2cc44a13f9b42de61a5ede3370a5fa29a390e"} err="failed to get container status \"c1896b4b544a8c43b11e19ce75d2cc44a13f9b42de61a5ede3370a5fa29a390e\": rpc error: code = NotFound desc = could not find container \"c1896b4b544a8c43b11e19ce75d2cc44a13f9b42de61a5ede3370a5fa29a390e\": container with ID starting with c1896b4b544a8c43b11e19ce75d2cc44a13f9b42de61a5ede3370a5fa29a390e not found: ID does not exist"
Oct 30 20:52:32.541762 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.541719    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-workload-socket\") pod \"7b0db74e-64b2-4bb2-83da-ee1669156580\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") "
Oct 30 20:52:32.541891 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.541810    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-token\" (UniqueName: \"kubernetes.io/projected/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-token\") pod \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") "
Oct 30 20:52:32.541891 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.541846    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-podinfo\") pod \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") "
Oct 30 20:52:32.541891 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.541864    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b0db74e-64b2-4bb2-83da-ee1669156580-kube-api-access-q7nhz\" (UniqueName: \"kubernetes.io/projected/7b0db74e-64b2-4bb2-83da-ee1669156580-kube-api-access-q7nhz\") pod \"7b0db74e-64b2-4bb2-83da-ee1669156580\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") "
Oct 30 20:52:32.542061 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.541902    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-envoy\") pod \"7b0db74e-64b2-4bb2-83da-ee1669156580\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") "
Oct 30 20:52:32.542061 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.541933    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-workload-certs\") pod \"7b0db74e-64b2-4bb2-83da-ee1669156580\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") "
Oct 30 20:52:32.542061 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.541954    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-data\") pod \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") "
Oct 30 20:52:32.542061 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.541968    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-workload-socket" pod "7b0db74e-64b2-4bb2-83da-ee1669156580" (UID: "7b0db74e-64b2-4bb2-83da-ee1669156580"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:32.542061 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.541981    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-token\" (UniqueName: \"kubernetes.io/projected/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-token\") pod \"7b0db74e-64b2-4bb2-83da-ee1669156580\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") "
Oct 30 20:52:32.542061 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.542046    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-credential-socket\") pod \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") "
Oct 30 20:52:32.542380 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.542116    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-data\") pod \"7b0db74e-64b2-4bb2-83da-ee1669156580\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") "
Oct 30 20:52:32.542380 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.542144    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-podinfo\") pod \"7b0db74e-64b2-4bb2-83da-ee1669156580\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") "
Oct 30 20:52:32.542380 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.542188    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19dfa0fc-a01f-495b-8d9f-e6e976bef433-kube-api-access-bh24t\" (UniqueName: \"kubernetes.io/projected/19dfa0fc-a01f-495b-8d9f-e6e976bef433-kube-api-access-bh24t\") pod \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") "
Oct 30 20:52:32.542380 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.542203    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-workload-certs" pod "7b0db74e-64b2-4bb2-83da-ee1669156580" (UID: "7b0db74e-64b2-4bb2-83da-ee1669156580"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:32.542380 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.542221    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-data" pod "19dfa0fc-a01f-495b-8d9f-e6e976bef433" (UID: "19dfa0fc-a01f-495b-8d9f-e6e976bef433"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:32.542380 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.542234    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istiod-ca-cert\") pod \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") "
Oct 30 20:52:32.542380 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.542361    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-envoy\") pod \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") "
Oct 30 20:52:32.542749 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.542407    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-workload-socket\") pod \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") "
Oct 30 20:52:32.542749 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.542463    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b0db74e-64b2-4bb2-83da-ee1669156580-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b0db74e-64b2-4bb2-83da-ee1669156580-istiod-ca-cert\") pod \"7b0db74e-64b2-4bb2-83da-ee1669156580\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") "
Oct 30 20:52:32.542749 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.542517    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-workload-certs\") pod \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\" (UID: \"19dfa0fc-a01f-495b-8d9f-e6e976bef433\") "
Oct 30 20:52:32.542749 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.542545    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-credential-socket\") pod \"7b0db74e-64b2-4bb2-83da-ee1669156580\" (UID: \"7b0db74e-64b2-4bb2-83da-ee1669156580\") "
Oct 30 20:52:32.543120 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.542357    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-credential-socket" pod "19dfa0fc-a01f-495b-8d9f-e6e976bef433" (UID: "19dfa0fc-a01f-495b-8d9f-e6e976bef433"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:32.543120 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.543098    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istiod-ca-cert" pod "19dfa0fc-a01f-495b-8d9f-e6e976bef433" (UID: "19dfa0fc-a01f-495b-8d9f-e6e976bef433"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:32.543443 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.543418    4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-workload-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.543543 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.543446    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-data" pod "7b0db74e-64b2-4bb2-83da-ee1669156580" (UID: "7b0db74e-64b2-4bb2-83da-ee1669156580"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:32.543543 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.543448    4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-workload-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.543543 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.543481    4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-data\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.543543 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.543499    4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-credential-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.543543 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.543515    4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istiod-ca-cert\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.543809 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.543639    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-workload-certs" pod "19dfa0fc-a01f-495b-8d9f-e6e976bef433" (UID: "19dfa0fc-a01f-495b-8d9f-e6e976bef433"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:32.543809 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.543730    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-workload-socket" pod "19dfa0fc-a01f-495b-8d9f-e6e976bef433" (UID: "19dfa0fc-a01f-495b-8d9f-e6e976bef433"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:32.543809 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.543762    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b0db74e-64b2-4bb2-83da-ee1669156580-istiod-ca-cert" pod "7b0db74e-64b2-4bb2-83da-ee1669156580" (UID: "7b0db74e-64b2-4bb2-83da-ee1669156580"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:32.543961 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.543850    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-credential-socket" pod "7b0db74e-64b2-4bb2-83da-ee1669156580" (UID: "7b0db74e-64b2-4bb2-83da-ee1669156580"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:32.544825 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.544766    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-podinfo" pod "19dfa0fc-a01f-495b-8d9f-e6e976bef433" (UID: "19dfa0fc-a01f-495b-8d9f-e6e976bef433"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:52:32.544937 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.544850    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-token" pod "7b0db74e-64b2-4bb2-83da-ee1669156580" (UID: "7b0db74e-64b2-4bb2-83da-ee1669156580"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:32.545243 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.545217    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-podinfo" pod "7b0db74e-64b2-4bb2-83da-ee1669156580" (UID: "7b0db74e-64b2-4bb2-83da-ee1669156580"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:52:32.545593 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.545568    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-envoy" pod "7b0db74e-64b2-4bb2-83da-ee1669156580" (UID: "7b0db74e-64b2-4bb2-83da-ee1669156580"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:32.545685 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.545621    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-token" pod "19dfa0fc-a01f-495b-8d9f-e6e976bef433" (UID: "19dfa0fc-a01f-495b-8d9f-e6e976bef433"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:32.545884 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.545862    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0db74e-64b2-4bb2-83da-ee1669156580-kube-api-access-q7nhz" pod "7b0db74e-64b2-4bb2-83da-ee1669156580" (UID: "7b0db74e-64b2-4bb2-83da-ee1669156580"). InnerVolumeSpecName "kube-api-access-q7nhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:32.545984 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.545968    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-envoy" pod "19dfa0fc-a01f-495b-8d9f-e6e976bef433" (UID: "19dfa0fc-a01f-495b-8d9f-e6e976bef433"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:32.546409 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.546394    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19dfa0fc-a01f-495b-8d9f-e6e976bef433-kube-api-access-bh24t" pod "19dfa0fc-a01f-495b-8d9f-e6e976bef433" (UID: "19dfa0fc-a01f-495b-8d9f-e6e976bef433"). InnerVolumeSpecName "kube-api-access-bh24t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:32.652702 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.652666    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-workload-socket\") pod \"165b506b-c809-43eb-a183-686869cf35db\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") "
Oct 30 20:52:32.652920 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.652716    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-istio-data\") pod \"165b506b-c809-43eb-a183-686869cf35db\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") "
Oct 30 20:52:32.652920 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.652773    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/165b506b-c809-43eb-a183-686869cf35db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165b506b-c809-43eb-a183-686869cf35db-istiod-ca-cert\") pod \"165b506b-c809-43eb-a183-686869cf35db\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") "
Oct 30 20:52:32.652920 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.652801    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-credential-socket\") pod \"165b506b-c809-43eb-a183-686869cf35db\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") "
Oct 30 20:52:32.652920 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.652838    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-istio-envoy\") pod \"165b506b-c809-43eb-a183-686869cf35db\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") "
Oct 30 20:52:32.652920 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.652914    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/165b506b-c809-43eb-a183-686869cf35db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165b506b-c809-43eb-a183-686869cf35db-istio-podinfo\") pod \"165b506b-c809-43eb-a183-686869cf35db\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") "
Oct 30 20:52:32.653185 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.652995    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-workload-socket" pod "165b506b-c809-43eb-a183-686869cf35db" (UID: "165b506b-c809-43eb-a183-686869cf35db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:32.653185 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.653073    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165b506b-c809-43eb-a183-686869cf35db-istio-token\" (UniqueName: \"kubernetes.io/projected/165b506b-c809-43eb-a183-686869cf35db-istio-token\") pod \"165b506b-c809-43eb-a183-686869cf35db\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") "
Oct 30 20:52:32.653185 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.653122    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165b506b-c809-43eb-a183-686869cf35db-kube-api-access-pmfl7\" (UniqueName: \"kubernetes.io/projected/165b506b-c809-43eb-a183-686869cf35db-kube-api-access-pmfl7\") pod \"165b506b-c809-43eb-a183-686869cf35db\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") "
Oct 30 20:52:32.653185 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.653157    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-workload-certs\") pod \"165b506b-c809-43eb-a183-686869cf35db\" (UID: \"165b506b-c809-43eb-a183-686869cf35db\") "
Oct 30 20:52:32.653488 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.653193    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-credential-socket" pod "165b506b-c809-43eb-a183-686869cf35db" (UID: "165b506b-c809-43eb-a183-686869cf35db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:32.653488 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.653249    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-istio-data" pod "165b506b-c809-43eb-a183-686869cf35db" (UID: "165b506b-c809-43eb-a183-686869cf35db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:32.653965 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.653655    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165b506b-c809-43eb-a183-686869cf35db-istiod-ca-cert" pod "165b506b-c809-43eb-a183-686869cf35db" (UID: "165b506b-c809-43eb-a183-686869cf35db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:32.653965 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.653830    4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-data\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.653965 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.653853    4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-podinfo\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.653965 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.653869    4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-workload-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.653965 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.653883    4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-istio-data\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.653965 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.653901    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh24t\" (UniqueName: \"kubernetes.io/projected/19dfa0fc-a01f-495b-8d9f-e6e976bef433-kube-api-access-bh24t\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.653965 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.653943    4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165b506b-c809-43eb-a183-686869cf35db-istiod-ca-cert\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.654498 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.654057    4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-credential-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.654498 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.654099    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-workload-certs" pod "165b506b-c809-43eb-a183-686869cf35db" (UID: "165b506b-c809-43eb-a183-686869cf35db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:32.654498 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.654109    4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-envoy\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.654498 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.654146    4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-workload-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.654498 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.654164    4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b0db74e-64b2-4bb2-83da-ee1669156580-istiod-ca-cert\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.654498 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.654180    4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19dfa0fc-a01f-495b-8d9f-e6e976bef433-workload-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.654498 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.654194    4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-credential-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.654498 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.654209    4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-token\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.654498 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.654222    4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19dfa0fc-a01f-495b-8d9f-e6e976bef433-istio-podinfo\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.654498 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.654235    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7nhz\" (UniqueName: \"kubernetes.io/projected/7b0db74e-64b2-4bb2-83da-ee1669156580-kube-api-access-q7nhz\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.654498 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.654250    4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-envoy\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.654498 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.654261    4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b0db74e-64b2-4bb2-83da-ee1669156580-istio-token\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.656027 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.656005    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-istio-envoy" pod "165b506b-c809-43eb-a183-686869cf35db" (UID: "165b506b-c809-43eb-a183-686869cf35db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:32.656097 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.656042    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165b506b-c809-43eb-a183-686869cf35db-istio-token" pod "165b506b-c809-43eb-a183-686869cf35db" (UID: "165b506b-c809-43eb-a183-686869cf35db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:32.656142 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.656130    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/165b506b-c809-43eb-a183-686869cf35db-istio-podinfo" pod "165b506b-c809-43eb-a183-686869cf35db" (UID: "165b506b-c809-43eb-a183-686869cf35db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:52:32.656466 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.656444    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165b506b-c809-43eb-a183-686869cf35db-kube-api-access-pmfl7" pod "165b506b-c809-43eb-a183-686869cf35db" (UID: "165b506b-c809-43eb-a183-686869cf35db"). InnerVolumeSpecName "kube-api-access-pmfl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:32.755380 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.755346    4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-istio-envoy\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.755380 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.755380    4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165b506b-c809-43eb-a183-686869cf35db-istio-podinfo\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.755572 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.755390    4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165b506b-c809-43eb-a183-686869cf35db-istio-token\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.755572 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.755399    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmfl7\" (UniqueName: \"kubernetes.io/projected/165b506b-c809-43eb-a183-686869cf35db-kube-api-access-pmfl7\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.755572 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.755411    4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165b506b-c809-43eb-a183-686869cf35db-workload-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:32.999740 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:32.999659    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"]
Oct 30 20:52:33.002610 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:33.002587    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-df031b-openshift-default-77cd44ff85-zf6tp"]
Oct 30 20:52:33.111416 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:33.111389    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"]
Oct 30 20:52:33.118153 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:33.118125    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-jpnpx"]
Oct 30 20:52:33.487742 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:33.487703    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw" event={"ID":"165b506b-c809-43eb-a183-686869cf35db","Type":"ContainerDied","Data":"2c734bc46c482a49741f8cb044a38227d743e770b33888664ac9ac65eca60982"}
Oct 30 20:52:33.487915 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:33.487749    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"
Oct 30 20:52:33.487915 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:33.487778    4471 scope.go:122] "RemoveContainer" containerID="402de59011fb87953260c8e332542a0e94fff69cb91b94ecb368dde240f1ae31"
Oct 30 20:52:33.651446 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:33.651413    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"]
Oct 30 20:52:33.655306 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:33.655283    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-f4tbw"]
Oct 30 20:52:34.432594 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:34.432564    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="165b506b-c809-43eb-a183-686869cf35db" path="/var/lib/kubelet/pods/165b506b-c809-43eb-a183-686869cf35db/volumes"
Oct 30 20:52:34.433044 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:34.433030    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19dfa0fc-a01f-495b-8d9f-e6e976bef433" path="/var/lib/kubelet/pods/19dfa0fc-a01f-495b-8d9f-e6e976bef433/volumes"
Oct 30 20:52:34.433465 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:34.433452    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b0db74e-64b2-4bb2-83da-ee1669156580" path="/var/lib/kubelet/pods/7b0db74e-64b2-4bb2-83da-ee1669156580/volumes"
Oct 30 20:52:35.015906 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.015871    4471 patch_prober.go:29] interesting pod/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:52:35.016090 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.015932    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7" podUID="d03eaf5f-735f-43cf-aa21-45901e9e6c04" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:52:35.503139 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.503109    4471 generic.go:356] "Generic (PLEG): container finished" podID="d03eaf5f-735f-43cf-aa21-45901e9e6c04" containerID="a9ba5e64b7f49212415bc458bd5f924389f7ad6b78f6b828640773f77becd359" exitCode=0
Oct 30 20:52:35.503490 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.503177    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7" event={"ID":"d03eaf5f-735f-43cf-aa21-45901e9e6c04","Type":"ContainerDied","Data":"a9ba5e64b7f49212415bc458bd5f924389f7ad6b78f6b828640773f77becd359"}
Oct 30 20:52:35.730727 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.730689    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:52:35.745716 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.745688    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:52:35.777571 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.777547    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:52:35.893569 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.893535    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-envoy\") pod \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") "
Oct 30 20:52:35.893793 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.893604    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-podinfo\") pod \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") "
Oct 30 20:52:35.893793 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.893641    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3613a886-6b08-4e25-807a-fbbe5c080775-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3613a886-6b08-4e25-807a-fbbe5c080775-istiod-ca-cert\") pod \"3613a886-6b08-4e25-807a-fbbe5c080775\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") "
Oct 30 20:52:35.893793 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.893669    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-workload-certs\") pod \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.893968    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-workload-certs" pod "d03eaf5f-735f-43cf-aa21-45901e9e6c04" (UID: "d03eaf5f-735f-43cf-aa21-45901e9e6c04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.894015    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-workload-socket\") pod \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.894040    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3613a886-6b08-4e25-807a-fbbe5c080775-istiod-ca-cert" pod "3613a886-6b08-4e25-807a-fbbe5c080775" (UID: "3613a886-6b08-4e25-807a-fbbe5c080775"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.894105    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-envoy\") pod \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.894137    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-data\") pod \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.894164    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e1b048-5b3f-43fd-b321-897f8d6e6e53-kube-api-access-ght5p\" (UniqueName: \"kubernetes.io/projected/03e1b048-5b3f-43fd-b321-897f8d6e6e53-kube-api-access-ght5p\") pod \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.894197    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-istio-data\") pod \"3613a886-6b08-4e25-807a-fbbe5c080775\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.894240    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-istio-envoy\") pod \"3613a886-6b08-4e25-807a-fbbe5c080775\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.894298    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-workload-socket" pod "d03eaf5f-735f-43cf-aa21-45901e9e6c04" (UID: "d03eaf5f-735f-43cf-aa21-45901e9e6c04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.894409    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-data" pod "03e1b048-5b3f-43fd-b321-897f8d6e6e53" (UID: "03e1b048-5b3f-43fd-b321-897f8d6e6e53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.894459    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3613a886-6b08-4e25-807a-fbbe5c080775-istio-token\" (UniqueName: \"kubernetes.io/projected/3613a886-6b08-4e25-807a-fbbe5c080775-istio-token\") pod \"3613a886-6b08-4e25-807a-fbbe5c080775\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.894495    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-workload-socket\") pod \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.894544    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-podinfo\") pod \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.894585    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istiod-ca-cert\") pod \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.894652    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-token\" (UniqueName: \"kubernetes.io/projected/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-token\") pod \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.894682    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istiod-ca-cert\") pod \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.894799    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-istio-data" pod "3613a886-6b08-4e25-807a-fbbe5c080775" (UID: "3613a886-6b08-4e25-807a-fbbe5c080775"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.894983    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-workload-socket" pod "03e1b048-5b3f-43fd-b321-897f8d6e6e53" (UID: "03e1b048-5b3f-43fd-b321-897f8d6e6e53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.895353    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istiod-ca-cert" pod "03e1b048-5b3f-43fd-b321-897f8d6e6e53" (UID: "03e1b048-5b3f-43fd-b321-897f8d6e6e53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.895418    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-workload-socket\") pod \"3613a886-6b08-4e25-807a-fbbe5c080775\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.895450    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3613a886-6b08-4e25-807a-fbbe5c080775-kube-api-access-x5lrg\" (UniqueName: \"kubernetes.io/projected/3613a886-6b08-4e25-807a-fbbe5c080775-kube-api-access-x5lrg\") pod \"3613a886-6b08-4e25-807a-fbbe5c080775\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.895762    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-workload-socket" pod "3613a886-6b08-4e25-807a-fbbe5c080775" (UID: "3613a886-6b08-4e25-807a-fbbe5c080775"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.895820    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-data\") pod \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.895838    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istiod-ca-cert" pod "d03eaf5f-735f-43cf-aa21-45901e9e6c04" (UID: "d03eaf5f-735f-43cf-aa21-45901e9e6c04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.895849    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-credential-socket\") pod \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.895940    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03eaf5f-735f-43cf-aa21-45901e9e6c04-kube-api-access-clplh\" (UniqueName: \"kubernetes.io/projected/d03eaf5f-735f-43cf-aa21-45901e9e6c04-kube-api-access-clplh\") pod \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.896097    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-credential-socket\") pod \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.896152    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-workload-certs\") pod \"3613a886-6b08-4e25-807a-fbbe5c080775\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.896180    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-data" pod "d03eaf5f-735f-43cf-aa21-45901e9e6c04" (UID: "d03eaf5f-735f-43cf-aa21-45901e9e6c04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.896196    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3613a886-6b08-4e25-807a-fbbe5c080775-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3613a886-6b08-4e25-807a-fbbe5c080775-istio-podinfo\") pod \"3613a886-6b08-4e25-807a-fbbe5c080775\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.896234    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-workload-certs\") pod \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\" (UID: \"03e1b048-5b3f-43fd-b321-897f8d6e6e53\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.896270    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-credential-socket\") pod \"3613a886-6b08-4e25-807a-fbbe5c080775\" (UID: \"3613a886-6b08-4e25-807a-fbbe5c080775\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.896300    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-token\" (UniqueName: \"kubernetes.io/projected/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-token\") pod \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\" (UID: \"d03eaf5f-735f-43cf-aa21-45901e9e6c04\") "
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.896727    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-workload-certs" pod "3613a886-6b08-4e25-807a-fbbe5c080775" (UID: "3613a886-6b08-4e25-807a-fbbe5c080775"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.897027    4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3613a886-6b08-4e25-807a-fbbe5c080775-istiod-ca-cert\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.897049    4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-workload-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.897068    4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-workload-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.897085    4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-data\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.897100    4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-istio-data\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.897113    4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-workload-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.897129    4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istiod-ca-cert\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.897144    4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istiod-ca-cert\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.897159    4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-workload-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.897177    4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-data\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.897166 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.897191    4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-workload-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.902238 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.897257    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-envoy" pod "03e1b048-5b3f-43fd-b321-897f8d6e6e53" (UID: "03e1b048-5b3f-43fd-b321-897f8d6e6e53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.902238 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.897304    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-credential-socket" pod "d03eaf5f-735f-43cf-aa21-45901e9e6c04" (UID: "d03eaf5f-735f-43cf-aa21-45901e9e6c04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.902238 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.897361    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-credential-socket" pod "03e1b048-5b3f-43fd-b321-897f8d6e6e53" (UID: "03e1b048-5b3f-43fd-b321-897f8d6e6e53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.902238 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.897756    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-workload-certs" pod "03e1b048-5b3f-43fd-b321-897f8d6e6e53" (UID: "03e1b048-5b3f-43fd-b321-897f8d6e6e53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.902238 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.897787    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-credential-socket" pod "3613a886-6b08-4e25-807a-fbbe5c080775" (UID: "3613a886-6b08-4e25-807a-fbbe5c080775"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.902238 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.899111    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-podinfo" pod "d03eaf5f-735f-43cf-aa21-45901e9e6c04" (UID: "d03eaf5f-735f-43cf-aa21-45901e9e6c04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:52:35.902238 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.900785    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-envoy" pod "d03eaf5f-735f-43cf-aa21-45901e9e6c04" (UID: "d03eaf5f-735f-43cf-aa21-45901e9e6c04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.902238 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.900851    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-istio-envoy" pod "3613a886-6b08-4e25-807a-fbbe5c080775" (UID: "3613a886-6b08-4e25-807a-fbbe5c080775"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:35.902238 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.900869    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-token" pod "03e1b048-5b3f-43fd-b321-897f8d6e6e53" (UID: "03e1b048-5b3f-43fd-b321-897f8d6e6e53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:35.902238 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.901286    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e1b048-5b3f-43fd-b321-897f8d6e6e53-kube-api-access-ght5p" pod "03e1b048-5b3f-43fd-b321-897f8d6e6e53" (UID: "03e1b048-5b3f-43fd-b321-897f8d6e6e53"). InnerVolumeSpecName "kube-api-access-ght5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:35.902238 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.901455    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3613a886-6b08-4e25-807a-fbbe5c080775-kube-api-access-x5lrg" pod "3613a886-6b08-4e25-807a-fbbe5c080775" (UID: "3613a886-6b08-4e25-807a-fbbe5c080775"). InnerVolumeSpecName "kube-api-access-x5lrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:35.902238 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.901499    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3613a886-6b08-4e25-807a-fbbe5c080775-istio-token" pod "3613a886-6b08-4e25-807a-fbbe5c080775" (UID: "3613a886-6b08-4e25-807a-fbbe5c080775"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:35.902238 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.901667    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3613a886-6b08-4e25-807a-fbbe5c080775-istio-podinfo" pod "3613a886-6b08-4e25-807a-fbbe5c080775" (UID: "3613a886-6b08-4e25-807a-fbbe5c080775"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:52:35.903027 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.902785    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03eaf5f-735f-43cf-aa21-45901e9e6c04-kube-api-access-clplh" pod "d03eaf5f-735f-43cf-aa21-45901e9e6c04" (UID: "d03eaf5f-735f-43cf-aa21-45901e9e6c04"). InnerVolumeSpecName "kube-api-access-clplh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:35.903027 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.902823    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-token" pod "d03eaf5f-735f-43cf-aa21-45901e9e6c04" (UID: "d03eaf5f-735f-43cf-aa21-45901e9e6c04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:35.903255 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.903237    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-podinfo" pod "03e1b048-5b3f-43fd-b321-897f8d6e6e53" (UID: "03e1b048-5b3f-43fd-b321-897f8d6e6e53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:52:35.998588 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.998557    4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-credential-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.998588 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.998584    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clplh\" (UniqueName: \"kubernetes.io/projected/d03eaf5f-735f-43cf-aa21-45901e9e6c04-kube-api-access-clplh\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.998588 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.998594    4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-credential-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.998807 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.998602    4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3613a886-6b08-4e25-807a-fbbe5c080775-istio-podinfo\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.998807 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.998611    4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-workload-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.998807 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.998621    4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-credential-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.998807 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.998629    4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-token\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.998807 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.998641    4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-envoy\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.998807 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.998653    4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-podinfo\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.998807 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.998661    4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d03eaf5f-735f-43cf-aa21-45901e9e6c04-istio-envoy\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.998807 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.998669    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ght5p\" (UniqueName: \"kubernetes.io/projected/03e1b048-5b3f-43fd-b321-897f8d6e6e53-kube-api-access-ght5p\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.998807 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.998677    4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3613a886-6b08-4e25-807a-fbbe5c080775-istio-envoy\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.998807 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.998685    4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3613a886-6b08-4e25-807a-fbbe5c080775-istio-token\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.998807 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.998693    4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-podinfo\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.998807 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.998701    4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e1b048-5b3f-43fd-b321-897f8d6e6e53-istio-token\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:35.998807 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:35.998708    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5lrg\" (UniqueName: \"kubernetes.io/projected/3613a886-6b08-4e25-807a-fbbe5c080775-kube-api-access-x5lrg\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:36.515397 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.515360    4471 generic.go:356] "Generic (PLEG): container finished" podID="3613a886-6b08-4e25-807a-fbbe5c080775" containerID="901cb1872378aa2a25d3b86cff3a747984eece3970a3ce8fce5c2af913b6395a" exitCode=0
Oct 30 20:52:36.515852 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.515428    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"
Oct 30 20:52:36.515852 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.515462    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c" event={"ID":"3613a886-6b08-4e25-807a-fbbe5c080775","Type":"ContainerDied","Data":"901cb1872378aa2a25d3b86cff3a747984eece3970a3ce8fce5c2af913b6395a"}
Oct 30 20:52:36.515852 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.515553    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c" event={"ID":"3613a886-6b08-4e25-807a-fbbe5c080775","Type":"ContainerDied","Data":"dc0b7a23bba98ac24bbfccfce031a8e810d75bf249858964c2d723ac7c6d5cb7"}
Oct 30 20:52:36.515852 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.515583    4471 scope.go:122] "RemoveContainer" containerID="901cb1872378aa2a25d3b86cff3a747984eece3970a3ce8fce5c2af913b6395a"
Oct 30 20:52:36.517255 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.517103    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"
Oct 30 20:52:36.517255 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.517135    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7" event={"ID":"d03eaf5f-735f-43cf-aa21-45901e9e6c04","Type":"ContainerDied","Data":"339beda8334f65873c99df4f37a594204baeaeb801095556764a78052c6c4d98"}
Oct 30 20:52:36.518923 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.518899    4471 generic.go:356] "Generic (PLEG): container finished" podID="03e1b048-5b3f-43fd-b321-897f8d6e6e53" containerID="cb0cfeec4ea49952fc4fa219e6e1137d91271b77ce04146f57e33e184851e66f" exitCode=0
Oct 30 20:52:36.519045 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.518967    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"
Oct 30 20:52:36.519201 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.518969    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr" event={"ID":"03e1b048-5b3f-43fd-b321-897f8d6e6e53","Type":"ContainerDied","Data":"cb0cfeec4ea49952fc4fa219e6e1137d91271b77ce04146f57e33e184851e66f"}
Oct 30 20:52:36.519305 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.519235    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr" event={"ID":"03e1b048-5b3f-43fd-b321-897f8d6e6e53","Type":"ContainerDied","Data":"7984b9c417c198ddde9f583323adfda8ede60e5328004bfd7d4572bacf6de649"}
Oct 30 20:52:36.528597 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.528561    4471 scope.go:122] "RemoveContainer" containerID="901cb1872378aa2a25d3b86cff3a747984eece3970a3ce8fce5c2af913b6395a"
Oct 30 20:52:36.528903 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:52:36.528883    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"901cb1872378aa2a25d3b86cff3a747984eece3970a3ce8fce5c2af913b6395a\": container with ID starting with 901cb1872378aa2a25d3b86cff3a747984eece3970a3ce8fce5c2af913b6395a not found: ID does not exist" containerID="901cb1872378aa2a25d3b86cff3a747984eece3970a3ce8fce5c2af913b6395a"
Oct 30 20:52:36.528975 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.528909    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"901cb1872378aa2a25d3b86cff3a747984eece3970a3ce8fce5c2af913b6395a"} err="failed to get container status \"901cb1872378aa2a25d3b86cff3a747984eece3970a3ce8fce5c2af913b6395a\": rpc error: code = NotFound desc = could not find container \"901cb1872378aa2a25d3b86cff3a747984eece3970a3ce8fce5c2af913b6395a\": container with ID starting with 901cb1872378aa2a25d3b86cff3a747984eece3970a3ce8fce5c2af913b6395a not found: ID does not exist"
Oct 30 20:52:36.528975 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.528927    4471 scope.go:122] "RemoveContainer" containerID="a9ba5e64b7f49212415bc458bd5f924389f7ad6b78f6b828640773f77becd359"
Oct 30 20:52:36.540898 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.540870    4471 scope.go:122] "RemoveContainer" containerID="cb0cfeec4ea49952fc4fa219e6e1137d91271b77ce04146f57e33e184851e66f"
Oct 30 20:52:36.550019 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.550002    4471 scope.go:122] "RemoveContainer" containerID="cb0cfeec4ea49952fc4fa219e6e1137d91271b77ce04146f57e33e184851e66f"
Oct 30 20:52:36.550283 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:52:36.550266    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0cfeec4ea49952fc4fa219e6e1137d91271b77ce04146f57e33e184851e66f\": container with ID starting with cb0cfeec4ea49952fc4fa219e6e1137d91271b77ce04146f57e33e184851e66f not found: ID does not exist" containerID="cb0cfeec4ea49952fc4fa219e6e1137d91271b77ce04146f57e33e184851e66f"
Oct 30 20:52:36.550364 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.550292    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0cfeec4ea49952fc4fa219e6e1137d91271b77ce04146f57e33e184851e66f"} err="failed to get container status \"cb0cfeec4ea49952fc4fa219e6e1137d91271b77ce04146f57e33e184851e66f\": rpc error: code = NotFound desc = could not find container \"cb0cfeec4ea49952fc4fa219e6e1137d91271b77ce04146f57e33e184851e66f\": container with ID starting with cb0cfeec4ea49952fc4fa219e6e1137d91271b77ce04146f57e33e184851e66f not found: ID does not exist"
Oct 30 20:52:36.680424 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.680387    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"]
Oct 30 20:52:36.728817 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.728783    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-qmp5c"]
Oct 30 20:52:36.790391 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.790357    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"]
Oct 30 20:52:36.793839 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.793814    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-4zwvr"]
Oct 30 20:52:36.901123 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.901089    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"]
Oct 30 20:52:36.904929 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:36.904905    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e2fc1-openshift-default-b5c99cf7f-k7tf7"]
Oct 30 20:52:38.433193 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:38.433164    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e1b048-5b3f-43fd-b321-897f8d6e6e53" path="/var/lib/kubelet/pods/03e1b048-5b3f-43fd-b321-897f8d6e6e53/volumes"
Oct 30 20:52:38.433627 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:38.433611    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3613a886-6b08-4e25-807a-fbbe5c080775" path="/var/lib/kubelet/pods/3613a886-6b08-4e25-807a-fbbe5c080775/volumes"
Oct 30 20:52:38.434015 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:38.434003    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d03eaf5f-735f-43cf-aa21-45901e9e6c04" path="/var/lib/kubelet/pods/d03eaf5f-735f-43cf-aa21-45901e9e6c04/volumes"
Oct 30 20:52:42.334858 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:42.334824    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"]
Oct 30 20:52:42.335412 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:42.335050    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng" podUID="b2d2f36f-2a1b-402b-8a71-08f174e078c4" containerName="istio-proxy" containerID="cri-o://cf7de9fd00ba72ca142b8b069bda4fd3bb487a0003569255e6545f98977dffe5" gracePeriod=30
Oct 30 20:52:47.653773 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.653498    4471 generic.go:356] "Generic (PLEG): container finished" podID="b2d2f36f-2a1b-402b-8a71-08f174e078c4" containerID="cf7de9fd00ba72ca142b8b069bda4fd3bb487a0003569255e6545f98977dffe5" exitCode=0
Oct 30 20:52:47.653773 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.653696    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng" event={"ID":"b2d2f36f-2a1b-402b-8a71-08f174e078c4","Type":"ContainerDied","Data":"cf7de9fd00ba72ca142b8b069bda4fd3bb487a0003569255e6545f98977dffe5"}
Oct 30 20:52:47.718150 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.718122    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:52:47.768782 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.768752    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-podinfo\") pod \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") "
Oct 30 20:52:47.768999 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.768790    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istiod-ca-cert\") pod \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") "
Oct 30 20:52:47.768999 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.768857    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2d2f36f-2a1b-402b-8a71-08f174e078c4-kube-api-access-x6dlm\" (UniqueName: \"kubernetes.io/projected/b2d2f36f-2a1b-402b-8a71-08f174e078c4-kube-api-access-x6dlm\") pod \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") "
Oct 30 20:52:47.768999 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.768924    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-token\" (UniqueName: \"kubernetes.io/projected/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-token\") pod \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") "
Oct 30 20:52:47.768999 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.768967    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-credential-socket\") pod \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") "
Oct 30 20:52:47.769228 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.769061    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-workload-certs\") pod \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") "
Oct 30 20:52:47.769228 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.769115    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-envoy\") pod \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") "
Oct 30 20:52:47.769228 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.769150    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-workload-socket\") pod \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") "
Oct 30 20:52:47.769228 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.769220    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-data\") pod \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\" (UID: \"b2d2f36f-2a1b-402b-8a71-08f174e078c4\") "
Oct 30 20:52:47.769458 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.769398    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istiod-ca-cert" pod "b2d2f36f-2a1b-402b-8a71-08f174e078c4" (UID: "b2d2f36f-2a1b-402b-8a71-08f174e078c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:52:47.770299 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.769629    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-data" pod "b2d2f36f-2a1b-402b-8a71-08f174e078c4" (UID: "b2d2f36f-2a1b-402b-8a71-08f174e078c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:47.770299 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.769965    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-credential-socket" pod "b2d2f36f-2a1b-402b-8a71-08f174e078c4" (UID: "b2d2f36f-2a1b-402b-8a71-08f174e078c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:47.770299 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.770017    4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-data\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:47.770299 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.770039    4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istiod-ca-cert\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:47.770299 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.769997    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-workload-socket" pod "b2d2f36f-2a1b-402b-8a71-08f174e078c4" (UID: "b2d2f36f-2a1b-402b-8a71-08f174e078c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:47.770299 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.770225    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-workload-certs" pod "b2d2f36f-2a1b-402b-8a71-08f174e078c4" (UID: "b2d2f36f-2a1b-402b-8a71-08f174e078c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:47.772280 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.772213    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-podinfo" pod "b2d2f36f-2a1b-402b-8a71-08f174e078c4" (UID: "b2d2f36f-2a1b-402b-8a71-08f174e078c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:52:47.775655 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.773022    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d2f36f-2a1b-402b-8a71-08f174e078c4-kube-api-access-x6dlm" pod "b2d2f36f-2a1b-402b-8a71-08f174e078c4" (UID: "b2d2f36f-2a1b-402b-8a71-08f174e078c4"). InnerVolumeSpecName "kube-api-access-x6dlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:47.775655 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.773066    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-token" pod "b2d2f36f-2a1b-402b-8a71-08f174e078c4" (UID: "b2d2f36f-2a1b-402b-8a71-08f174e078c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:52:47.775655 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.773166    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-envoy" pod "b2d2f36f-2a1b-402b-8a71-08f174e078c4" (UID: "b2d2f36f-2a1b-402b-8a71-08f174e078c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:52:47.871753 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.871718    4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-workload-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:47.871753 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.871751    4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-envoy\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:47.871753 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.871762    4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-workload-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:47.872026 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.871772    4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-podinfo\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:47.872026 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.871786    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6dlm\" (UniqueName: \"kubernetes.io/projected/b2d2f36f-2a1b-402b-8a71-08f174e078c4-kube-api-access-x6dlm\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:47.872026 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.871802    4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2d2f36f-2a1b-402b-8a71-08f174e078c4-istio-token\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:47.872026 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:47.871814    4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2d2f36f-2a1b-402b-8a71-08f174e078c4-credential-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:52:48.661106 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:48.661070    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng" event={"ID":"b2d2f36f-2a1b-402b-8a71-08f174e078c4","Type":"ContainerDied","Data":"c7672ab18a035a6cf9e5af1bef6385cab8665f203c31990f2dc08e0d4df2f195"}
Oct 30 20:52:48.661545 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:48.661109    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"
Oct 30 20:52:48.661545 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:48.661144    4471 scope.go:122] "RemoveContainer" containerID="cf7de9fd00ba72ca142b8b069bda4fd3bb487a0003569255e6545f98977dffe5"
Oct 30 20:52:48.918118 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:48.918083    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"]
Oct 30 20:52:48.951280 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:48.951247    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-z68ng"]
Oct 30 20:52:50.432757 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:52:50.432726    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2d2f36f-2a1b-402b-8a71-08f174e078c4" path="/var/lib/kubelet/pods/b2d2f36f-2a1b-402b-8a71-08f174e078c4/volumes"
Oct 30 20:53:00.954065 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:53:00.954028    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:53:14.134649 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:14.134612    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc"]
Oct 30 20:53:14.245775 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:14.245740    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg"]
Oct 30 20:53:19.136338 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.136276    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc" podUID="f1bf6069-9c41-4f32-83c6-91561d7833b4" containerName="payload-processing" containerID="cri-o://c06fc3f444ae8103b5517f4e352dd6906bbbce380a45f72db00fbe88fa0d949d" gracePeriod=25
Oct 30 20:53:19.246794 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.246750    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg" podUID="2a7a1c0a-2a90-487e-9a94-780475a94fb5" containerName="payload-pre-processing" containerID="cri-o://dc9077c6115590b6b77e3374ab1f380edf4c6672984fd17effe8cdd2be8874e9" gracePeriod=25
Oct 30 20:53:19.404953 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.404879    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc"
Oct 30 20:53:19.491384 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.491351    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1bf6069-9c41-4f32-83c6-91561d7833b4-kube-api-access-kn47m\" (UniqueName: \"kubernetes.io/projected/f1bf6069-9c41-4f32-83c6-91561d7833b4-kube-api-access-kn47m\") pod \"f1bf6069-9c41-4f32-83c6-91561d7833b4\" (UID: \"f1bf6069-9c41-4f32-83c6-91561d7833b4\") "
Oct 30 20:53:19.491578 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.491399    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1bf6069-9c41-4f32-83c6-91561d7833b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1bf6069-9c41-4f32-83c6-91561d7833b4-plugins-config-volume\") pod \"f1bf6069-9c41-4f32-83c6-91561d7833b4\" (UID: \"f1bf6069-9c41-4f32-83c6-91561d7833b4\") "
Oct 30 20:53:19.491823 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.491797    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1bf6069-9c41-4f32-83c6-91561d7833b4-plugins-config-volume" pod "f1bf6069-9c41-4f32-83c6-91561d7833b4" (UID: "f1bf6069-9c41-4f32-83c6-91561d7833b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:19.493856 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.493825    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1bf6069-9c41-4f32-83c6-91561d7833b4-kube-api-access-kn47m" pod "f1bf6069-9c41-4f32-83c6-91561d7833b4" (UID: "f1bf6069-9c41-4f32-83c6-91561d7833b4"). InnerVolumeSpecName "kube-api-access-kn47m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:19.511680 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.511660    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg"
Oct 30 20:53:19.593197 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.593167    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a7a1c0a-2a90-487e-9a94-780475a94fb5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7a1c0a-2a90-487e-9a94-780475a94fb5-plugins-config-volume\") pod \"2a7a1c0a-2a90-487e-9a94-780475a94fb5\" (UID: \"2a7a1c0a-2a90-487e-9a94-780475a94fb5\") "
Oct 30 20:53:19.593393 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.593274    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7a1c0a-2a90-487e-9a94-780475a94fb5-kube-api-access-whmpq\" (UniqueName: \"kubernetes.io/projected/2a7a1c0a-2a90-487e-9a94-780475a94fb5-kube-api-access-whmpq\") pod \"2a7a1c0a-2a90-487e-9a94-780475a94fb5\" (UID: \"2a7a1c0a-2a90-487e-9a94-780475a94fb5\") "
Oct 30 20:53:19.593573 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.593547    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a7a1c0a-2a90-487e-9a94-780475a94fb5-plugins-config-volume" pod "2a7a1c0a-2a90-487e-9a94-780475a94fb5" (UID: "2a7a1c0a-2a90-487e-9a94-780475a94fb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:19.593668 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.593641    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn47m\" (UniqueName: \"kubernetes.io/projected/f1bf6069-9c41-4f32-83c6-91561d7833b4-kube-api-access-kn47m\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:19.593668 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.593659    4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1bf6069-9c41-4f32-83c6-91561d7833b4-plugins-config-volume\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:19.595493 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.595467    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7a1c0a-2a90-487e-9a94-780475a94fb5-kube-api-access-whmpq" pod "2a7a1c0a-2a90-487e-9a94-780475a94fb5" (UID: "2a7a1c0a-2a90-487e-9a94-780475a94fb5"). InnerVolumeSpecName "kube-api-access-whmpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:19.694903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.694818    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whmpq\" (UniqueName: \"kubernetes.io/projected/2a7a1c0a-2a90-487e-9a94-780475a94fb5-kube-api-access-whmpq\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:19.694903 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.694845    4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a7a1c0a-2a90-487e-9a94-780475a94fb5-plugins-config-volume\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:19.851699 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.851656    4471 generic.go:356] "Generic (PLEG): container finished" podID="f1bf6069-9c41-4f32-83c6-91561d7833b4" containerID="c06fc3f444ae8103b5517f4e352dd6906bbbce380a45f72db00fbe88fa0d949d" exitCode=0
Oct 30 20:53:19.851885 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.851727    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc"
Oct 30 20:53:19.851885 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.851736    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc" event={"ID":"f1bf6069-9c41-4f32-83c6-91561d7833b4","Type":"ContainerDied","Data":"c06fc3f444ae8103b5517f4e352dd6906bbbce380a45f72db00fbe88fa0d949d"}
Oct 30 20:53:19.851885 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.851815    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc" event={"ID":"f1bf6069-9c41-4f32-83c6-91561d7833b4","Type":"ContainerDied","Data":"dd35beb89d5454433ddf17a3516f2693b46c9e4fa5ecbaab5db234722145e96c"}
Oct 30 20:53:19.851885 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.851842    4471 scope.go:122] "RemoveContainer" containerID="c06fc3f444ae8103b5517f4e352dd6906bbbce380a45f72db00fbe88fa0d949d"
Oct 30 20:53:19.853465 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.853439    4471 generic.go:356] "Generic (PLEG): container finished" podID="2a7a1c0a-2a90-487e-9a94-780475a94fb5" containerID="dc9077c6115590b6b77e3374ab1f380edf4c6672984fd17effe8cdd2be8874e9" exitCode=0
Oct 30 20:53:19.853599 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.853470    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg" event={"ID":"2a7a1c0a-2a90-487e-9a94-780475a94fb5","Type":"ContainerDied","Data":"dc9077c6115590b6b77e3374ab1f380edf4c6672984fd17effe8cdd2be8874e9"}
Oct 30 20:53:19.853599 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.853510    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg" event={"ID":"2a7a1c0a-2a90-487e-9a94-780475a94fb5","Type":"ContainerDied","Data":"0dbb5d4658c394b7ce298d784e6bf85683add6780df0fda8aa61fe49f4427965"}
Oct 30 20:53:19.853599 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.853529    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg"
Oct 30 20:53:19.862953 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.862932    4471 scope.go:122] "RemoveContainer" containerID="c06fc3f444ae8103b5517f4e352dd6906bbbce380a45f72db00fbe88fa0d949d"
Oct 30 20:53:19.863268 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:53:19.863249    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c06fc3f444ae8103b5517f4e352dd6906bbbce380a45f72db00fbe88fa0d949d\": container with ID starting with c06fc3f444ae8103b5517f4e352dd6906bbbce380a45f72db00fbe88fa0d949d not found: ID does not exist" containerID="c06fc3f444ae8103b5517f4e352dd6906bbbce380a45f72db00fbe88fa0d949d"
Oct 30 20:53:19.863362 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.863276    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06fc3f444ae8103b5517f4e352dd6906bbbce380a45f72db00fbe88fa0d949d"} err="failed to get container status \"c06fc3f444ae8103b5517f4e352dd6906bbbce380a45f72db00fbe88fa0d949d\": rpc error: code = NotFound desc = could not find container \"c06fc3f444ae8103b5517f4e352dd6906bbbce380a45f72db00fbe88fa0d949d\": container with ID starting with c06fc3f444ae8103b5517f4e352dd6906bbbce380a45f72db00fbe88fa0d949d not found: ID does not exist"
Oct 30 20:53:19.863362 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.863294    4471 scope.go:122] "RemoveContainer" containerID="dc9077c6115590b6b77e3374ab1f380edf4c6672984fd17effe8cdd2be8874e9"
Oct 30 20:53:19.872795 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.872776    4471 scope.go:122] "RemoveContainer" containerID="dc9077c6115590b6b77e3374ab1f380edf4c6672984fd17effe8cdd2be8874e9"
Oct 30 20:53:19.873083 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:53:19.873061    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc9077c6115590b6b77e3374ab1f380edf4c6672984fd17effe8cdd2be8874e9\": container with ID starting with dc9077c6115590b6b77e3374ab1f380edf4c6672984fd17effe8cdd2be8874e9 not found: ID does not exist" containerID="dc9077c6115590b6b77e3374ab1f380edf4c6672984fd17effe8cdd2be8874e9"
Oct 30 20:53:19.873134 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:19.873091    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc9077c6115590b6b77e3374ab1f380edf4c6672984fd17effe8cdd2be8874e9"} err="failed to get container status \"dc9077c6115590b6b77e3374ab1f380edf4c6672984fd17effe8cdd2be8874e9\": rpc error: code = NotFound desc = could not find container \"dc9077c6115590b6b77e3374ab1f380edf4c6672984fd17effe8cdd2be8874e9\": container with ID starting with dc9077c6115590b6b77e3374ab1f380edf4c6672984fd17effe8cdd2be8874e9 not found: ID does not exist"
Oct 30 20:53:20.015051 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:20.014958    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc"]
Oct 30 20:53:20.064014 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:20.063979    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3d13a-d7f5b8d76-4sjfc"]
Oct 30 20:53:20.125772 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:20.125738    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg"]
Oct 30 20:53:20.130010 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:20.129985    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3d13a-799f746d9f-29rcg"]
Oct 30 20:53:20.431901 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:20.431866    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a7a1c0a-2a90-487e-9a94-780475a94fb5" path="/var/lib/kubelet/pods/2a7a1c0a-2a90-487e-9a94-780475a94fb5/volumes"
Oct 30 20:53:20.432288 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:20.432280    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1bf6069-9c41-4f32-83c6-91561d7833b4" path="/var/lib/kubelet/pods/f1bf6069-9c41-4f32-83c6-91561d7833b4/volumes"
Oct 30 20:53:30.955351 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:53:30.955247    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:53:35.316523 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:35.316435    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"]
Oct 30 20:53:35.316979 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:35.316700    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g" podUID="0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9" containerName="maas-api" containerID="cri-o://e14690aca0b0165197ae643eb25ac11d55d13d2726ba985d35026e309474c056" gracePeriod=30
Oct 30 20:53:35.661915 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:35.661889    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"
Oct 30 20:53:35.809791 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:35.809753    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-maas-api-tls\") pod \"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9\" (UID: \"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9\") "
Oct 30 20:53:35.809981 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:35.809851    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-openshift-service-ca\") pod \"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9\" (UID: \"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9\") "
Oct 30 20:53:35.810052 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:35.810006    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-kube-api-access-sf4lq\" (UniqueName: \"kubernetes.io/projected/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-kube-api-access-sf4lq\") pod \"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9\" (UID: \"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9\") "
Oct 30 20:53:35.810122 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:35.810093    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-metrics-certs\") pod \"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9\" (UID: \"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9\") "
Oct 30 20:53:35.810241 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:35.810215    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-openshift-service-ca" pod "0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9" (UID: "0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:35.810749 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:35.810725    4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-openshift-service-ca\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:35.813567 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:35.813524    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-kube-api-access-sf4lq" pod "0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9" (UID: "0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9"). InnerVolumeSpecName "kube-api-access-sf4lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:35.814231 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:35.814204    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-maas-api-tls" pod "0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9" (UID: "0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:53:35.814362 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:35.814310    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-metrics-certs" pod "0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9" (UID: "0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:53:35.923826 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:35.923638    4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-maas-api-tls\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:35.923826 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:35.923677    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf4lq\" (UniqueName: \"kubernetes.io/projected/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-kube-api-access-sf4lq\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:35.923826 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:35.923693    4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9-metrics-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:35.949363 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:35.949310    4471 generic.go:356] "Generic (PLEG): container finished" podID="0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9" containerID="e14690aca0b0165197ae643eb25ac11d55d13d2726ba985d35026e309474c056" exitCode=0
Oct 30 20:53:35.949536 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:35.949379    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g" event={"ID":"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9","Type":"ContainerDied","Data":"e14690aca0b0165197ae643eb25ac11d55d13d2726ba985d35026e309474c056"}
Oct 30 20:53:35.949536 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:35.949399    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"
Oct 30 20:53:35.949536 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:35.949450    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g" event={"ID":"0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9","Type":"ContainerDied","Data":"f996f35008fa774700ea101c19109378c7f76390324f6b3e304aa4ef36da4c20"}
Oct 30 20:53:35.949536 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:35.949503    4471 scope.go:122] "RemoveContainer" containerID="e14690aca0b0165197ae643eb25ac11d55d13d2726ba985d35026e309474c056"
Oct 30 20:53:35.959437 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:35.959415    4471 scope.go:122] "RemoveContainer" containerID="e14690aca0b0165197ae643eb25ac11d55d13d2726ba985d35026e309474c056"
Oct 30 20:53:35.959676 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:53:35.959659    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e14690aca0b0165197ae643eb25ac11d55d13d2726ba985d35026e309474c056\": container with ID starting with e14690aca0b0165197ae643eb25ac11d55d13d2726ba985d35026e309474c056 not found: ID does not exist" containerID="e14690aca0b0165197ae643eb25ac11d55d13d2726ba985d35026e309474c056"
Oct 30 20:53:35.959745 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:35.959683    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e14690aca0b0165197ae643eb25ac11d55d13d2726ba985d35026e309474c056"} err="failed to get container status \"e14690aca0b0165197ae643eb25ac11d55d13d2726ba985d35026e309474c056\": rpc error: code = NotFound desc = could not find container \"e14690aca0b0165197ae643eb25ac11d55d13d2726ba985d35026e309474c056\": container with ID starting with e14690aca0b0165197ae643eb25ac11d55d13d2726ba985d35026e309474c056 not found: ID does not exist"
Oct 30 20:53:36.116347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:36.116288    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"]
Oct 30 20:53:36.125461 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:36.125434    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-0ac4e5-6dcb659-62k7g"]
Oct 30 20:53:36.432526 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:36.432463    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9" path="/var/lib/kubelet/pods/0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9/volumes"
Oct 30 20:53:36.493678 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:36.493645    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh"]
Oct 30 20:53:41.494043 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:41.493989    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh" podUID="84057aab-629f-4918-9d85-cd3e2d3a95b4" containerName="payload-pre-processing" containerID="cri-o://b004aefefe6b5748bc835f26c1b7ee69d1487997deb22362f087ba748bbaebcf" gracePeriod=25
Oct 30 20:53:41.752969 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:41.752944    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh"
Oct 30 20:53:41.918709 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:41.918672    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84057aab-629f-4918-9d85-cd3e2d3a95b4-kube-api-access-sdhfh\" (UniqueName: \"kubernetes.io/projected/84057aab-629f-4918-9d85-cd3e2d3a95b4-kube-api-access-sdhfh\") pod \"84057aab-629f-4918-9d85-cd3e2d3a95b4\" (UID: \"84057aab-629f-4918-9d85-cd3e2d3a95b4\") "
Oct 30 20:53:41.918925 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:41.918905    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84057aab-629f-4918-9d85-cd3e2d3a95b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84057aab-629f-4918-9d85-cd3e2d3a95b4-plugins-config-volume\") pod \"84057aab-629f-4918-9d85-cd3e2d3a95b4\" (UID: \"84057aab-629f-4918-9d85-cd3e2d3a95b4\") "
Oct 30 20:53:41.919568 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:41.919523    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84057aab-629f-4918-9d85-cd3e2d3a95b4-plugins-config-volume" pod "84057aab-629f-4918-9d85-cd3e2d3a95b4" (UID: "84057aab-629f-4918-9d85-cd3e2d3a95b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:41.926347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:41.925522    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84057aab-629f-4918-9d85-cd3e2d3a95b4-kube-api-access-sdhfh" pod "84057aab-629f-4918-9d85-cd3e2d3a95b4" (UID: "84057aab-629f-4918-9d85-cd3e2d3a95b4"). InnerVolumeSpecName "kube-api-access-sdhfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:42.000574 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:42.000533    4471 generic.go:356] "Generic (PLEG): container finished" podID="84057aab-629f-4918-9d85-cd3e2d3a95b4" containerID="b004aefefe6b5748bc835f26c1b7ee69d1487997deb22362f087ba748bbaebcf" exitCode=0
Oct 30 20:53:42.000756 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:42.000584    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh"
Oct 30 20:53:42.000756 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:42.000615    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh" event={"ID":"84057aab-629f-4918-9d85-cd3e2d3a95b4","Type":"ContainerDied","Data":"b004aefefe6b5748bc835f26c1b7ee69d1487997deb22362f087ba748bbaebcf"}
Oct 30 20:53:42.000756 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:42.000673    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh" event={"ID":"84057aab-629f-4918-9d85-cd3e2d3a95b4","Type":"ContainerDied","Data":"b1636e70c00d6d611330a36343b82859dab8ccaea7030ab2328989328ca0f708"}
Oct 30 20:53:42.000756 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:42.000703    4471 scope.go:122] "RemoveContainer" containerID="b004aefefe6b5748bc835f26c1b7ee69d1487997deb22362f087ba748bbaebcf"
Oct 30 20:53:42.012107 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:42.011765    4471 scope.go:122] "RemoveContainer" containerID="b004aefefe6b5748bc835f26c1b7ee69d1487997deb22362f087ba748bbaebcf"
Oct 30 20:53:42.012208 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:53:42.012102    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b004aefefe6b5748bc835f26c1b7ee69d1487997deb22362f087ba748bbaebcf\": container with ID starting with b004aefefe6b5748bc835f26c1b7ee69d1487997deb22362f087ba748bbaebcf not found: ID does not exist" containerID="b004aefefe6b5748bc835f26c1b7ee69d1487997deb22362f087ba748bbaebcf"
Oct 30 20:53:42.012208 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:42.012135    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b004aefefe6b5748bc835f26c1b7ee69d1487997deb22362f087ba748bbaebcf"} err="failed to get container status \"b004aefefe6b5748bc835f26c1b7ee69d1487997deb22362f087ba748bbaebcf\": rpc error: code = NotFound desc = could not find container \"b004aefefe6b5748bc835f26c1b7ee69d1487997deb22362f087ba748bbaebcf\": container with ID starting with b004aefefe6b5748bc835f26c1b7ee69d1487997deb22362f087ba748bbaebcf not found: ID does not exist"
Oct 30 20:53:42.021005 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:42.020986    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdhfh\" (UniqueName: \"kubernetes.io/projected/84057aab-629f-4918-9d85-cd3e2d3a95b4-kube-api-access-sdhfh\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:42.021085 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:42.021007    4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84057aab-629f-4918-9d85-cd3e2d3a95b4-plugins-config-volume\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:42.178376 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:42.178264    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh"]
Oct 30 20:53:42.182475 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:42.182447    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3f09ec-cc86cc579-sp9kh"]
Oct 30 20:53:42.434604 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:42.434524    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84057aab-629f-4918-9d85-cd3e2d3a95b4" path="/var/lib/kubelet/pods/84057aab-629f-4918-9d85-cd3e2d3a95b4/volumes"
Oct 30 20:53:51.391915 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:51.391880    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"]
Oct 30 20:53:51.392473 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:51.392097    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm" podUID="c7b255dd-817d-480d-9f58-df362bca182d" containerName="istio-proxy" containerID="cri-o://16218d06bc5d19172fc894ba0fdb105c0914a0820087a85fd54a876faec2a3ad" gracePeriod=30
Oct 30 20:53:56.649489 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.649464    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:53:56.721723 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.721639    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b255dd-817d-480d-9f58-df362bca182d-kube-api-access-x6g66\" (UniqueName: \"kubernetes.io/projected/c7b255dd-817d-480d-9f58-df362bca182d-kube-api-access-x6g66\") pod \"c7b255dd-817d-480d-9f58-df362bca182d\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") "
Oct 30 20:53:56.721723 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.721676    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-credential-socket\") pod \"c7b255dd-817d-480d-9f58-df362bca182d\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") "
Oct 30 20:53:56.721932 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.721879    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-workload-certs\") pod \"c7b255dd-817d-480d-9f58-df362bca182d\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") "
Oct 30 20:53:56.721988 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.721946    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b255dd-817d-480d-9f58-df362bca182d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b255dd-817d-480d-9f58-df362bca182d-istiod-ca-cert\") pod \"c7b255dd-817d-480d-9f58-df362bca182d\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") "
Oct 30 20:53:56.721988 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.721950    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-credential-socket" pod "c7b255dd-817d-480d-9f58-df362bca182d" (UID: "c7b255dd-817d-480d-9f58-df362bca182d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:56.721988 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.721982    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7b255dd-817d-480d-9f58-df362bca182d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b255dd-817d-480d-9f58-df362bca182d-istio-podinfo\") pod \"c7b255dd-817d-480d-9f58-df362bca182d\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") "
Oct 30 20:53:56.722143 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.722026    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b255dd-817d-480d-9f58-df362bca182d-istio-token\" (UniqueName: \"kubernetes.io/projected/c7b255dd-817d-480d-9f58-df362bca182d-istio-token\") pod \"c7b255dd-817d-480d-9f58-df362bca182d\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") "
Oct 30 20:53:56.722143 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.722102    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-workload-certs" pod "c7b255dd-817d-480d-9f58-df362bca182d" (UID: "c7b255dd-817d-480d-9f58-df362bca182d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:56.722143 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.722108    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-istio-envoy\") pod \"c7b255dd-817d-480d-9f58-df362bca182d\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") "
Oct 30 20:53:56.722299 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.722197    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-workload-socket\") pod \"c7b255dd-817d-480d-9f58-df362bca182d\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") "
Oct 30 20:53:56.722387 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.722317    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b255dd-817d-480d-9f58-df362bca182d-istiod-ca-cert" pod "c7b255dd-817d-480d-9f58-df362bca182d" (UID: "c7b255dd-817d-480d-9f58-df362bca182d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:56.722598 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.722572    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-istio-data\") pod \"c7b255dd-817d-480d-9f58-df362bca182d\" (UID: \"c7b255dd-817d-480d-9f58-df362bca182d\") "
Oct 30 20:53:56.722767 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.722699    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-workload-socket" pod "c7b255dd-817d-480d-9f58-df362bca182d" (UID: "c7b255dd-817d-480d-9f58-df362bca182d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:56.722842 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.722813    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-istio-data" pod "c7b255dd-817d-480d-9f58-df362bca182d" (UID: "c7b255dd-817d-480d-9f58-df362bca182d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:56.723529 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.723507    4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b255dd-817d-480d-9f58-df362bca182d-istiod-ca-cert\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:56.723628 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.723535    4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-workload-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:56.723628 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.723550    4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-istio-data\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:56.723628 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.723564    4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-credential-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:56.723628 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.723577    4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-workload-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:56.724376 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.724349    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7b255dd-817d-480d-9f58-df362bca182d-istio-podinfo" pod "c7b255dd-817d-480d-9f58-df362bca182d" (UID: "c7b255dd-817d-480d-9f58-df362bca182d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:53:56.724550 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.724527    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b255dd-817d-480d-9f58-df362bca182d-istio-token" pod "c7b255dd-817d-480d-9f58-df362bca182d" (UID: "c7b255dd-817d-480d-9f58-df362bca182d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:56.724797 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.724779    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-istio-envoy" pod "c7b255dd-817d-480d-9f58-df362bca182d" (UID: "c7b255dd-817d-480d-9f58-df362bca182d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:53:56.724850 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.724829    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b255dd-817d-480d-9f58-df362bca182d-kube-api-access-x6g66" pod "c7b255dd-817d-480d-9f58-df362bca182d" (UID: "c7b255dd-817d-480d-9f58-df362bca182d"). InnerVolumeSpecName "kube-api-access-x6g66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:56.828350 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.824407    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6g66\" (UniqueName: \"kubernetes.io/projected/c7b255dd-817d-480d-9f58-df362bca182d-kube-api-access-x6g66\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:56.828350 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.824446    4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b255dd-817d-480d-9f58-df362bca182d-istio-podinfo\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:56.828350 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.824460    4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b255dd-817d-480d-9f58-df362bca182d-istio-token\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:56.828350 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:56.824474    4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b255dd-817d-480d-9f58-df362bca182d-istio-envoy\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:57.140848 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:57.140817    4471 generic.go:356] "Generic (PLEG): container finished" podID="c7b255dd-817d-480d-9f58-df362bca182d" containerID="16218d06bc5d19172fc894ba0fdb105c0914a0820087a85fd54a876faec2a3ad" exitCode=0
Oct 30 20:53:57.141061 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:57.140879    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"
Oct 30 20:53:57.141061 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:57.140897    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm" event={"ID":"c7b255dd-817d-480d-9f58-df362bca182d","Type":"ContainerDied","Data":"16218d06bc5d19172fc894ba0fdb105c0914a0820087a85fd54a876faec2a3ad"}
Oct 30 20:53:57.141061 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:57.140964    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm" event={"ID":"c7b255dd-817d-480d-9f58-df362bca182d","Type":"ContainerDied","Data":"9867a2e4d709b417e6597ede78383b6a7629634cb54c66cab6284ed48ccb429b"}
Oct 30 20:53:57.141061 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:57.140994    4471 scope.go:122] "RemoveContainer" containerID="16218d06bc5d19172fc894ba0fdb105c0914a0820087a85fd54a876faec2a3ad"
Oct 30 20:53:57.151305 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:57.151283    4471 scope.go:122] "RemoveContainer" containerID="2b4be582624ab4776007fa8c03a7350b7fa73b819fd32e2029bcd6b5f024b14c"
Oct 30 20:53:57.163098 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:57.162870    4471 scope.go:122] "RemoveContainer" containerID="16218d06bc5d19172fc894ba0fdb105c0914a0820087a85fd54a876faec2a3ad"
Oct 30 20:53:57.163587 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:53:57.163555    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16218d06bc5d19172fc894ba0fdb105c0914a0820087a85fd54a876faec2a3ad\": container with ID starting with 16218d06bc5d19172fc894ba0fdb105c0914a0820087a85fd54a876faec2a3ad not found: ID does not exist" containerID="16218d06bc5d19172fc894ba0fdb105c0914a0820087a85fd54a876faec2a3ad"
Oct 30 20:53:57.163685 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:57.163592    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16218d06bc5d19172fc894ba0fdb105c0914a0820087a85fd54a876faec2a3ad"} err="failed to get container status \"16218d06bc5d19172fc894ba0fdb105c0914a0820087a85fd54a876faec2a3ad\": rpc error: code = NotFound desc = could not find container \"16218d06bc5d19172fc894ba0fdb105c0914a0820087a85fd54a876faec2a3ad\": container with ID starting with 16218d06bc5d19172fc894ba0fdb105c0914a0820087a85fd54a876faec2a3ad not found: ID does not exist"
Oct 30 20:53:57.163685 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:57.163613    4471 scope.go:122] "RemoveContainer" containerID="2b4be582624ab4776007fa8c03a7350b7fa73b819fd32e2029bcd6b5f024b14c"
Oct 30 20:53:57.163867 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:53:57.163847    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b4be582624ab4776007fa8c03a7350b7fa73b819fd32e2029bcd6b5f024b14c\": container with ID starting with 2b4be582624ab4776007fa8c03a7350b7fa73b819fd32e2029bcd6b5f024b14c not found: ID does not exist" containerID="2b4be582624ab4776007fa8c03a7350b7fa73b819fd32e2029bcd6b5f024b14c"
Oct 30 20:53:57.163927 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:57.163873    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b4be582624ab4776007fa8c03a7350b7fa73b819fd32e2029bcd6b5f024b14c"} err="failed to get container status \"2b4be582624ab4776007fa8c03a7350b7fa73b819fd32e2029bcd6b5f024b14c\": rpc error: code = NotFound desc = could not find container \"2b4be582624ab4776007fa8c03a7350b7fa73b819fd32e2029bcd6b5f024b14c\": container with ID starting with 2b4be582624ab4776007fa8c03a7350b7fa73b819fd32e2029bcd6b5f024b14c not found: ID does not exist"
Oct 30 20:53:57.330970 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:57.330939    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"]
Oct 30 20:53:57.380188 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:57.380158    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-99c030-openshift-default-7949dd7947-crhnm"]
Oct 30 20:53:57.839567 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:57.839532    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6"]
Oct 30 20:53:58.433256 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:53:58.433222    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b255dd-817d-480d-9f58-df362bca182d" path="/var/lib/kubelet/pods/c7b255dd-817d-480d-9f58-df362bca182d/volumes"
Oct 30 20:54:00.954664 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:54:00.954621    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:54:02.840248 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:02.840185    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6" podUID="e9dd5e0a-7d52-4c72-a274-348b97f5dcf1" containerName="payload-pre-processing" containerID="cri-o://c4e93d428f1cb035b3726de430351fa9678a6918d9ce55f25197efcbd5990037" gracePeriod=25
Oct 30 20:54:03.096572 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:03.096507    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6"
Oct 30 20:54:03.153623 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:03.153597    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9dd5e0a-7d52-4c72-a274-348b97f5dcf1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9dd5e0a-7d52-4c72-a274-348b97f5dcf1-plugins-config-volume\") pod \"e9dd5e0a-7d52-4c72-a274-348b97f5dcf1\" (UID: \"e9dd5e0a-7d52-4c72-a274-348b97f5dcf1\") "
Oct 30 20:54:03.153794 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:03.153762    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9dd5e0a-7d52-4c72-a274-348b97f5dcf1-kube-api-access-xztmr\" (UniqueName: \"kubernetes.io/projected/e9dd5e0a-7d52-4c72-a274-348b97f5dcf1-kube-api-access-xztmr\") pod \"e9dd5e0a-7d52-4c72-a274-348b97f5dcf1\" (UID: \"e9dd5e0a-7d52-4c72-a274-348b97f5dcf1\") "
Oct 30 20:54:03.154023 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:03.153998    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9dd5e0a-7d52-4c72-a274-348b97f5dcf1-plugins-config-volume" pod "e9dd5e0a-7d52-4c72-a274-348b97f5dcf1" (UID: "e9dd5e0a-7d52-4c72-a274-348b97f5dcf1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:03.154236 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:03.154220    4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9dd5e0a-7d52-4c72-a274-348b97f5dcf1-plugins-config-volume\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:03.155950 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:03.155930    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9dd5e0a-7d52-4c72-a274-348b97f5dcf1-kube-api-access-xztmr" pod "e9dd5e0a-7d52-4c72-a274-348b97f5dcf1" (UID: "e9dd5e0a-7d52-4c72-a274-348b97f5dcf1"). InnerVolumeSpecName "kube-api-access-xztmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:03.209824 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:03.209784    4471 generic.go:356] "Generic (PLEG): container finished" podID="e9dd5e0a-7d52-4c72-a274-348b97f5dcf1" containerID="c4e93d428f1cb035b3726de430351fa9678a6918d9ce55f25197efcbd5990037" exitCode=0
Oct 30 20:54:03.209978 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:03.209823    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6" event={"ID":"e9dd5e0a-7d52-4c72-a274-348b97f5dcf1","Type":"ContainerDied","Data":"c4e93d428f1cb035b3726de430351fa9678a6918d9ce55f25197efcbd5990037"}
Oct 30 20:54:03.209978 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:03.209838    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6"
Oct 30 20:54:03.209978 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:03.209886    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6" event={"ID":"e9dd5e0a-7d52-4c72-a274-348b97f5dcf1","Type":"ContainerDied","Data":"144137d827b690e1e7807187146c7eb7b03dba56a8e1557b464a1957fb6f317b"}
Oct 30 20:54:03.209978 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:03.209910    4471 scope.go:122] "RemoveContainer" containerID="c4e93d428f1cb035b3726de430351fa9678a6918d9ce55f25197efcbd5990037"
Oct 30 20:54:03.223070 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:03.223048    4471 scope.go:122] "RemoveContainer" containerID="c4e93d428f1cb035b3726de430351fa9678a6918d9ce55f25197efcbd5990037"
Oct 30 20:54:03.223453 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:54:03.223424    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e93d428f1cb035b3726de430351fa9678a6918d9ce55f25197efcbd5990037\": container with ID starting with c4e93d428f1cb035b3726de430351fa9678a6918d9ce55f25197efcbd5990037 not found: ID does not exist" containerID="c4e93d428f1cb035b3726de430351fa9678a6918d9ce55f25197efcbd5990037"
Oct 30 20:54:03.223567 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:03.223460    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e93d428f1cb035b3726de430351fa9678a6918d9ce55f25197efcbd5990037"} err="failed to get container status \"c4e93d428f1cb035b3726de430351fa9678a6918d9ce55f25197efcbd5990037\": rpc error: code = NotFound desc = could not find container \"c4e93d428f1cb035b3726de430351fa9678a6918d9ce55f25197efcbd5990037\": container with ID starting with c4e93d428f1cb035b3726de430351fa9678a6918d9ce55f25197efcbd5990037 not found: ID does not exist"
Oct 30 20:54:03.255598 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:03.255558    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xztmr\" (UniqueName: \"kubernetes.io/projected/e9dd5e0a-7d52-4c72-a274-348b97f5dcf1-kube-api-access-xztmr\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:03.374356 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:03.374314    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6"]
Oct 30 20:54:03.378402 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:03.378374    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab9c38-6989695fdc-khzb6"]
Oct 30 20:54:03.724682 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:03.724596    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx"]
Oct 30 20:54:03.833170 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:03.833122    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2"]
Oct 30 20:54:04.433185 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:04.433152    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9dd5e0a-7d52-4c72-a274-348b97f5dcf1" path="/var/lib/kubelet/pods/e9dd5e0a-7d52-4c72-a274-348b97f5dcf1/volumes"
Oct 30 20:54:08.725701 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:08.725649    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx" podUID="c121a634-e346-4be2-872d-fc943d82ad37" containerName="payload-processing" containerID="cri-o://ff6e98c71b7a43006361600d34f2fd6d2aba47c745eed1889664f830ae8cfc14" gracePeriod=25
Oct 30 20:54:08.834287 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:08.834225    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2" podUID="e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12" containerName="payload-pre-processing" containerID="cri-o://97e2f89a97cb9b83888a7f649893bfafc686db82d9ab9a1b665a5bf2fb3d0f2c" gracePeriod=25
Oct 30 20:54:08.997426 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:08.997359    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx"
Oct 30 20:54:09.096860 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.096827    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2"
Oct 30 20:54:09.141025 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.140986    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c121a634-e346-4be2-872d-fc943d82ad37-kube-api-access-tfh4w\" (UniqueName: \"kubernetes.io/projected/c121a634-e346-4be2-872d-fc943d82ad37-kube-api-access-tfh4w\") pod \"c121a634-e346-4be2-872d-fc943d82ad37\" (UID: \"c121a634-e346-4be2-872d-fc943d82ad37\") "
Oct 30 20:54:09.141205 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.141114    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c121a634-e346-4be2-872d-fc943d82ad37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c121a634-e346-4be2-872d-fc943d82ad37-plugins-config-volume\") pod \"c121a634-e346-4be2-872d-fc943d82ad37\" (UID: \"c121a634-e346-4be2-872d-fc943d82ad37\") "
Oct 30 20:54:09.141565 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.141539    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c121a634-e346-4be2-872d-fc943d82ad37-plugins-config-volume" pod "c121a634-e346-4be2-872d-fc943d82ad37" (UID: "c121a634-e346-4be2-872d-fc943d82ad37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:09.143373 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.143347    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c121a634-e346-4be2-872d-fc943d82ad37-kube-api-access-tfh4w" pod "c121a634-e346-4be2-872d-fc943d82ad37" (UID: "c121a634-e346-4be2-872d-fc943d82ad37"). InnerVolumeSpecName "kube-api-access-tfh4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:09.242641 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.242613    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12-plugins-config-volume\") pod \"e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12\" (UID: \"e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12\") "
Oct 30 20:54:09.242817 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.242705    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12-kube-api-access-fpccx\" (UniqueName: \"kubernetes.io/projected/e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12-kube-api-access-fpccx\") pod \"e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12\" (UID: \"e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12\") "
Oct 30 20:54:09.243055 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.243031    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12-plugins-config-volume" pod "e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12" (UID: "e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:09.243100 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.243060    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfh4w\" (UniqueName: \"kubernetes.io/projected/c121a634-e346-4be2-872d-fc943d82ad37-kube-api-access-tfh4w\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:09.243100 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.243085    4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c121a634-e346-4be2-872d-fc943d82ad37-plugins-config-volume\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:09.244969 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.244945    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12-kube-api-access-fpccx" pod "e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12" (UID: "e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12"). InnerVolumeSpecName "kube-api-access-fpccx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:09.315833 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.297224    4471 generic.go:356] "Generic (PLEG): container finished" podID="e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12" containerID="97e2f89a97cb9b83888a7f649893bfafc686db82d9ab9a1b665a5bf2fb3d0f2c" exitCode=0
Oct 30 20:54:09.315833 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.297450    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2" event={"ID":"e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12","Type":"ContainerDied","Data":"97e2f89a97cb9b83888a7f649893bfafc686db82d9ab9a1b665a5bf2fb3d0f2c"}
Oct 30 20:54:09.315833 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.297540    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2" event={"ID":"e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12","Type":"ContainerDied","Data":"579fcc078c29854514271c20409f70c13d34f707503fc8872d6612167b177a43"}
Oct 30 20:54:09.315833 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.297602    4471 scope.go:122] "RemoveContainer" containerID="97e2f89a97cb9b83888a7f649893bfafc686db82d9ab9a1b665a5bf2fb3d0f2c"
Oct 30 20:54:09.315833 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.299631    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2"
Oct 30 20:54:09.352384 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.351225    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpccx\" (UniqueName: \"kubernetes.io/projected/e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12-kube-api-access-fpccx\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:09.352384 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.351251    4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12-plugins-config-volume\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:09.353102 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.353067    4471 generic.go:356] "Generic (PLEG): container finished" podID="c121a634-e346-4be2-872d-fc943d82ad37" containerID="ff6e98c71b7a43006361600d34f2fd6d2aba47c745eed1889664f830ae8cfc14" exitCode=0
Oct 30 20:54:09.353236 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.353172    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx" event={"ID":"c121a634-e346-4be2-872d-fc943d82ad37","Type":"ContainerDied","Data":"ff6e98c71b7a43006361600d34f2fd6d2aba47c745eed1889664f830ae8cfc14"}
Oct 30 20:54:09.353236 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.353226    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx" event={"ID":"c121a634-e346-4be2-872d-fc943d82ad37","Type":"ContainerDied","Data":"58e28e333e2f31990f81a1cbc5633e222e1adfed8914809b3c5ec26ec7fcf7ef"}
Oct 30 20:54:09.354655 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.353311    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx"
Oct 30 20:54:09.360078 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.360059    4471 scope.go:122] "RemoveContainer" containerID="97e2f89a97cb9b83888a7f649893bfafc686db82d9ab9a1b665a5bf2fb3d0f2c"
Oct 30 20:54:09.360434 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:54:09.360412    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97e2f89a97cb9b83888a7f649893bfafc686db82d9ab9a1b665a5bf2fb3d0f2c\": container with ID starting with 97e2f89a97cb9b83888a7f649893bfafc686db82d9ab9a1b665a5bf2fb3d0f2c not found: ID does not exist" containerID="97e2f89a97cb9b83888a7f649893bfafc686db82d9ab9a1b665a5bf2fb3d0f2c"
Oct 30 20:54:09.360502 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.360444    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e2f89a97cb9b83888a7f649893bfafc686db82d9ab9a1b665a5bf2fb3d0f2c"} err="failed to get container status \"97e2f89a97cb9b83888a7f649893bfafc686db82d9ab9a1b665a5bf2fb3d0f2c\": rpc error: code = NotFound desc = could not find container \"97e2f89a97cb9b83888a7f649893bfafc686db82d9ab9a1b665a5bf2fb3d0f2c\": container with ID starting with 97e2f89a97cb9b83888a7f649893bfafc686db82d9ab9a1b665a5bf2fb3d0f2c not found: ID does not exist"
Oct 30 20:54:09.360502 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.360462    4471 scope.go:122] "RemoveContainer" containerID="ff6e98c71b7a43006361600d34f2fd6d2aba47c745eed1889664f830ae8cfc14"
Oct 30 20:54:09.369620 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.369585    4471 scope.go:122] "RemoveContainer" containerID="ff6e98c71b7a43006361600d34f2fd6d2aba47c745eed1889664f830ae8cfc14"
Oct 30 20:54:09.369918 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:54:09.369897    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff6e98c71b7a43006361600d34f2fd6d2aba47c745eed1889664f830ae8cfc14\": container with ID starting with ff6e98c71b7a43006361600d34f2fd6d2aba47c745eed1889664f830ae8cfc14 not found: ID does not exist" containerID="ff6e98c71b7a43006361600d34f2fd6d2aba47c745eed1889664f830ae8cfc14"
Oct 30 20:54:09.369982 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.369926    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff6e98c71b7a43006361600d34f2fd6d2aba47c745eed1889664f830ae8cfc14"} err="failed to get container status \"ff6e98c71b7a43006361600d34f2fd6d2aba47c745eed1889664f830ae8cfc14\": rpc error: code = NotFound desc = could not find container \"ff6e98c71b7a43006361600d34f2fd6d2aba47c745eed1889664f830ae8cfc14\": container with ID starting with ff6e98c71b7a43006361600d34f2fd6d2aba47c745eed1889664f830ae8cfc14 not found: ID does not exist"
Oct 30 20:54:09.498434 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.498400    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2"]
Oct 30 20:54:09.505767 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.505739    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8ac60-68f54d8959-pj2v2"]
Oct 30 20:54:09.622805 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.622713    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx"]
Oct 30 20:54:09.638642 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:09.638616    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f8ac60-65845bcc7d-528mx"]
Oct 30 20:54:10.432820 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:10.432787    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c121a634-e346-4be2-872d-fc943d82ad37" path="/var/lib/kubelet/pods/c121a634-e346-4be2-872d-fc943d82ad37/volumes"
Oct 30 20:54:10.433215 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:10.433191    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12" path="/var/lib/kubelet/pods/e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12/volumes"
Oct 30 20:54:30.955373 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:54:30.955310    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:54:42.722347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:42.718548    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"]
Oct 30 20:54:42.722347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:42.718859    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv" podUID="a2199190-0e92-4238-8011-84a7d273a53e" containerName="maas-api" containerID="cri-o://3f019ea9d03df5a06b88171744df8f4fd98d0849d7fc7b166732e9d1e28c2123" gracePeriod=30
Oct 30 20:54:43.046110 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:43.046082    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"
Oct 30 20:54:43.172351 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:43.171785    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2199190-0e92-4238-8011-84a7d273a53e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2199190-0e92-4238-8011-84a7d273a53e-metrics-certs\") pod \"a2199190-0e92-4238-8011-84a7d273a53e\" (UID: \"a2199190-0e92-4238-8011-84a7d273a53e\") "
Oct 30 20:54:43.172351 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:43.171842    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2199190-0e92-4238-8011-84a7d273a53e-kube-api-access-tqxsm\" (UniqueName: \"kubernetes.io/projected/a2199190-0e92-4238-8011-84a7d273a53e-kube-api-access-tqxsm\") pod \"a2199190-0e92-4238-8011-84a7d273a53e\" (UID: \"a2199190-0e92-4238-8011-84a7d273a53e\") "
Oct 30 20:54:43.172351 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:43.171885    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2199190-0e92-4238-8011-84a7d273a53e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2199190-0e92-4238-8011-84a7d273a53e-maas-api-tls\") pod \"a2199190-0e92-4238-8011-84a7d273a53e\" (UID: \"a2199190-0e92-4238-8011-84a7d273a53e\") "
Oct 30 20:54:43.172351 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:43.171994    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2199190-0e92-4238-8011-84a7d273a53e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2199190-0e92-4238-8011-84a7d273a53e-openshift-service-ca\") pod \"a2199190-0e92-4238-8011-84a7d273a53e\" (UID: \"a2199190-0e92-4238-8011-84a7d273a53e\") "
Oct 30 20:54:43.177069 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:43.172877    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2199190-0e92-4238-8011-84a7d273a53e-openshift-service-ca" pod "a2199190-0e92-4238-8011-84a7d273a53e" (UID: "a2199190-0e92-4238-8011-84a7d273a53e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:43.177518 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:43.177464    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2199190-0e92-4238-8011-84a7d273a53e-metrics-certs" pod "a2199190-0e92-4238-8011-84a7d273a53e" (UID: "a2199190-0e92-4238-8011-84a7d273a53e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:54:43.183994 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:43.177861    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2199190-0e92-4238-8011-84a7d273a53e-kube-api-access-tqxsm" pod "a2199190-0e92-4238-8011-84a7d273a53e" (UID: "a2199190-0e92-4238-8011-84a7d273a53e"). InnerVolumeSpecName "kube-api-access-tqxsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:43.205345 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:43.202431    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2199190-0e92-4238-8011-84a7d273a53e-maas-api-tls" pod "a2199190-0e92-4238-8011-84a7d273a53e" (UID: "a2199190-0e92-4238-8011-84a7d273a53e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:54:43.273475 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:43.273395    4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2199190-0e92-4238-8011-84a7d273a53e-metrics-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:43.273475 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:43.273425    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqxsm\" (UniqueName: \"kubernetes.io/projected/a2199190-0e92-4238-8011-84a7d273a53e-kube-api-access-tqxsm\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:43.273475 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:43.273435    4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2199190-0e92-4238-8011-84a7d273a53e-maas-api-tls\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:43.273475 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:43.273449    4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2199190-0e92-4238-8011-84a7d273a53e-openshift-service-ca\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:43.615813 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:43.615726    4471 generic.go:356] "Generic (PLEG): container finished" podID="a2199190-0e92-4238-8011-84a7d273a53e" containerID="3f019ea9d03df5a06b88171744df8f4fd98d0849d7fc7b166732e9d1e28c2123" exitCode=0
Oct 30 20:54:43.615813 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:43.615777    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"
Oct 30 20:54:43.616045 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:43.615822    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv" event={"ID":"a2199190-0e92-4238-8011-84a7d273a53e","Type":"ContainerDied","Data":"3f019ea9d03df5a06b88171744df8f4fd98d0849d7fc7b166732e9d1e28c2123"}
Oct 30 20:54:43.616045 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:43.615874    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv" event={"ID":"a2199190-0e92-4238-8011-84a7d273a53e","Type":"ContainerDied","Data":"852238d92c22cd906c14e72ccbb1dbbfeecd2d9ed9fb324f52e40edeb11d3994"}
Oct 30 20:54:43.616045 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:43.615894    4471 scope.go:122] "RemoveContainer" containerID="3f019ea9d03df5a06b88171744df8f4fd98d0849d7fc7b166732e9d1e28c2123"
Oct 30 20:54:43.626881 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:43.626859    4471 scope.go:122] "RemoveContainer" containerID="3f019ea9d03df5a06b88171744df8f4fd98d0849d7fc7b166732e9d1e28c2123"
Oct 30 20:54:43.628094 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:54:43.627368    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f019ea9d03df5a06b88171744df8f4fd98d0849d7fc7b166732e9d1e28c2123\": container with ID starting with 3f019ea9d03df5a06b88171744df8f4fd98d0849d7fc7b166732e9d1e28c2123 not found: ID does not exist" containerID="3f019ea9d03df5a06b88171744df8f4fd98d0849d7fc7b166732e9d1e28c2123"
Oct 30 20:54:43.628094 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:43.627423    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f019ea9d03df5a06b88171744df8f4fd98d0849d7fc7b166732e9d1e28c2123"} err="failed to get container status \"3f019ea9d03df5a06b88171744df8f4fd98d0849d7fc7b166732e9d1e28c2123\": rpc error: code = NotFound desc = could not find container \"3f019ea9d03df5a06b88171744df8f4fd98d0849d7fc7b166732e9d1e28c2123\": container with ID starting with 3f019ea9d03df5a06b88171744df8f4fd98d0849d7fc7b166732e9d1e28c2123 not found: ID does not exist"
Oct 30 20:54:43.812363 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:43.799253    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"]
Oct 30 20:54:43.812363 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:43.810745    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3f09ec-748867b8cd-rb2hv"]
Oct 30 20:54:44.431575 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:54:44.431540    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2199190-0e92-4238-8011-84a7d273a53e" path="/var/lib/kubelet/pods/a2199190-0e92-4238-8011-84a7d273a53e/volumes"
Oct 30 20:55:00.954798 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:55:00.954710    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:55:02.923822 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:02.923780    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4"]
Oct 30 20:55:03.035352 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:03.035285    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz"]
Oct 30 20:55:07.924912 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:07.924868    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4" podUID="67597528-00c9-4a1d-a5ed-02a170390a40" containerName="payload-processing" containerID="cri-o://cf883912f9e50d568fb5d02c424d9e3d0c25b306a2a80c9650303adc4dc91b31" gracePeriod=25
Oct 30 20:55:08.036242 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.036200    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz" podUID="4d739d8d-97fe-4c71-a9f4-d8d72ea853b6" containerName="payload-pre-processing" containerID="cri-o://f51c66cb102a3d8b6f4b5a432f4b74dbdb2e3d1f00ae3ff0c28b8b0815034ef8" gracePeriod=25
Oct 30 20:55:08.188200 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.188134    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4"
Oct 30 20:55:08.262908 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.262865    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67597528-00c9-4a1d-a5ed-02a170390a40-kube-api-access-7pdfp\" (UniqueName: \"kubernetes.io/projected/67597528-00c9-4a1d-a5ed-02a170390a40-kube-api-access-7pdfp\") pod \"67597528-00c9-4a1d-a5ed-02a170390a40\" (UID: \"67597528-00c9-4a1d-a5ed-02a170390a40\") "
Oct 30 20:55:08.263096 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.263058    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67597528-00c9-4a1d-a5ed-02a170390a40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67597528-00c9-4a1d-a5ed-02a170390a40-plugins-config-volume\") pod \"67597528-00c9-4a1d-a5ed-02a170390a40\" (UID: \"67597528-00c9-4a1d-a5ed-02a170390a40\") "
Oct 30 20:55:08.263784 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.263749    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67597528-00c9-4a1d-a5ed-02a170390a40-plugins-config-volume" pod "67597528-00c9-4a1d-a5ed-02a170390a40" (UID: "67597528-00c9-4a1d-a5ed-02a170390a40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:08.266639 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.266606    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67597528-00c9-4a1d-a5ed-02a170390a40-kube-api-access-7pdfp" pod "67597528-00c9-4a1d-a5ed-02a170390a40" (UID: "67597528-00c9-4a1d-a5ed-02a170390a40"). InnerVolumeSpecName "kube-api-access-7pdfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:08.298076 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.298054    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz"
Oct 30 20:55:08.368446 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.365169    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d739d8d-97fe-4c71-a9f4-d8d72ea853b6-kube-api-access-nkzdp\" (UniqueName: \"kubernetes.io/projected/4d739d8d-97fe-4c71-a9f4-d8d72ea853b6-kube-api-access-nkzdp\") pod \"4d739d8d-97fe-4c71-a9f4-d8d72ea853b6\" (UID: \"4d739d8d-97fe-4c71-a9f4-d8d72ea853b6\") "
Oct 30 20:55:08.368446 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.365384    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d739d8d-97fe-4c71-a9f4-d8d72ea853b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d739d8d-97fe-4c71-a9f4-d8d72ea853b6-plugins-config-volume\") pod \"4d739d8d-97fe-4c71-a9f4-d8d72ea853b6\" (UID: \"4d739d8d-97fe-4c71-a9f4-d8d72ea853b6\") "
Oct 30 20:55:08.368446 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.366058    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pdfp\" (UniqueName: \"kubernetes.io/projected/67597528-00c9-4a1d-a5ed-02a170390a40-kube-api-access-7pdfp\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:08.368446 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.366081    4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67597528-00c9-4a1d-a5ed-02a170390a40-plugins-config-volume\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:08.368446 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.366514    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d739d8d-97fe-4c71-a9f4-d8d72ea853b6-plugins-config-volume" pod "4d739d8d-97fe-4c71-a9f4-d8d72ea853b6" (UID: "4d739d8d-97fe-4c71-a9f4-d8d72ea853b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:08.374063 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.374031    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d739d8d-97fe-4c71-a9f4-d8d72ea853b6-kube-api-access-nkzdp" pod "4d739d8d-97fe-4c71-a9f4-d8d72ea853b6" (UID: "4d739d8d-97fe-4c71-a9f4-d8d72ea853b6"). InnerVolumeSpecName "kube-api-access-nkzdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:08.467206 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.467173    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkzdp\" (UniqueName: \"kubernetes.io/projected/4d739d8d-97fe-4c71-a9f4-d8d72ea853b6-kube-api-access-nkzdp\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:08.467393 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.467210    4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d739d8d-97fe-4c71-a9f4-d8d72ea853b6-plugins-config-volume\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:08.789207 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.789124    4471 generic.go:356] "Generic (PLEG): container finished" podID="67597528-00c9-4a1d-a5ed-02a170390a40" containerID="cf883912f9e50d568fb5d02c424d9e3d0c25b306a2a80c9650303adc4dc91b31" exitCode=0
Oct 30 20:55:08.789207 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.789186    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4"
Oct 30 20:55:08.789458 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.789205    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4" event={"ID":"67597528-00c9-4a1d-a5ed-02a170390a40","Type":"ContainerDied","Data":"cf883912f9e50d568fb5d02c424d9e3d0c25b306a2a80c9650303adc4dc91b31"}
Oct 30 20:55:08.789458 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.789255    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4" event={"ID":"67597528-00c9-4a1d-a5ed-02a170390a40","Type":"ContainerDied","Data":"f1357445968b4e8a84606da9c4a462726a529f15196ef3d04d2cb021749175fe"}
Oct 30 20:55:08.789458 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.789283    4471 scope.go:122] "RemoveContainer" containerID="cf883912f9e50d568fb5d02c424d9e3d0c25b306a2a80c9650303adc4dc91b31"
Oct 30 20:55:08.791044 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.791016    4471 generic.go:356] "Generic (PLEG): container finished" podID="4d739d8d-97fe-4c71-a9f4-d8d72ea853b6" containerID="f51c66cb102a3d8b6f4b5a432f4b74dbdb2e3d1f00ae3ff0c28b8b0815034ef8" exitCode=0
Oct 30 20:55:08.791154 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.791075    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz"
Oct 30 20:55:08.791206 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.791075    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz" event={"ID":"4d739d8d-97fe-4c71-a9f4-d8d72ea853b6","Type":"ContainerDied","Data":"f51c66cb102a3d8b6f4b5a432f4b74dbdb2e3d1f00ae3ff0c28b8b0815034ef8"}
Oct 30 20:55:08.791265 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.791221    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz" event={"ID":"4d739d8d-97fe-4c71-a9f4-d8d72ea853b6","Type":"ContainerDied","Data":"e138c992cbecc03a6d989dea9d6482aed372ffc5d52125bbcffee3bf0987fa9c"}
Oct 30 20:55:08.799148 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.799128    4471 scope.go:122] "RemoveContainer" containerID="cf883912f9e50d568fb5d02c424d9e3d0c25b306a2a80c9650303adc4dc91b31"
Oct 30 20:55:08.799444 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:55:08.799424    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf883912f9e50d568fb5d02c424d9e3d0c25b306a2a80c9650303adc4dc91b31\": container with ID starting with cf883912f9e50d568fb5d02c424d9e3d0c25b306a2a80c9650303adc4dc91b31 not found: ID does not exist" containerID="cf883912f9e50d568fb5d02c424d9e3d0c25b306a2a80c9650303adc4dc91b31"
Oct 30 20:55:08.799495 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.799454    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf883912f9e50d568fb5d02c424d9e3d0c25b306a2a80c9650303adc4dc91b31"} err="failed to get container status \"cf883912f9e50d568fb5d02c424d9e3d0c25b306a2a80c9650303adc4dc91b31\": rpc error: code = NotFound desc = could not find container \"cf883912f9e50d568fb5d02c424d9e3d0c25b306a2a80c9650303adc4dc91b31\": container with ID starting with cf883912f9e50d568fb5d02c424d9e3d0c25b306a2a80c9650303adc4dc91b31 not found: ID does not exist"
Oct 30 20:55:08.799495 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.799472    4471 scope.go:122] "RemoveContainer" containerID="f51c66cb102a3d8b6f4b5a432f4b74dbdb2e3d1f00ae3ff0c28b8b0815034ef8"
Oct 30 20:55:08.807820 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.807799    4471 scope.go:122] "RemoveContainer" containerID="f51c66cb102a3d8b6f4b5a432f4b74dbdb2e3d1f00ae3ff0c28b8b0815034ef8"
Oct 30 20:55:08.808108 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:55:08.808087    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f51c66cb102a3d8b6f4b5a432f4b74dbdb2e3d1f00ae3ff0c28b8b0815034ef8\": container with ID starting with f51c66cb102a3d8b6f4b5a432f4b74dbdb2e3d1f00ae3ff0c28b8b0815034ef8 not found: ID does not exist" containerID="f51c66cb102a3d8b6f4b5a432f4b74dbdb2e3d1f00ae3ff0c28b8b0815034ef8"
Oct 30 20:55:08.808184 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.808119    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f51c66cb102a3d8b6f4b5a432f4b74dbdb2e3d1f00ae3ff0c28b8b0815034ef8"} err="failed to get container status \"f51c66cb102a3d8b6f4b5a432f4b74dbdb2e3d1f00ae3ff0c28b8b0815034ef8\": rpc error: code = NotFound desc = could not find container \"f51c66cb102a3d8b6f4b5a432f4b74dbdb2e3d1f00ae3ff0c28b8b0815034ef8\": container with ID starting with f51c66cb102a3d8b6f4b5a432f4b74dbdb2e3d1f00ae3ff0c28b8b0815034ef8 not found: ID does not exist"
Oct 30 20:55:08.951921 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.951883    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz"]
Oct 30 20:55:08.954594 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:08.954573    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-712099-789b9796b7-8gkkz"]
Oct 30 20:55:09.073455 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:09.071974    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4"]
Oct 30 20:55:09.082829 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:09.082797    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-712099-7949d459d4-qcbm4"]
Oct 30 20:55:10.228946 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:10.228912    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng"]
Oct 30 20:55:10.431861 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:10.431830    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d739d8d-97fe-4c71-a9f4-d8d72ea853b6" path="/var/lib/kubelet/pods/4d739d8d-97fe-4c71-a9f4-d8d72ea853b6/volumes"
Oct 30 20:55:10.432207 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:10.432194    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67597528-00c9-4a1d-a5ed-02a170390a40" path="/var/lib/kubelet/pods/67597528-00c9-4a1d-a5ed-02a170390a40/volumes"
Oct 30 20:55:11.583960 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:11.583923    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"]
Oct 30 20:55:11.584372 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:11.584103    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx" podUID="ec075762-7e92-471c-bc35-c456ab936ad0" containerName="maas-api" containerID="cri-o://1f771057e49da49e2ccb9499f4089dbf7ffcb70fe9b050b74beaa33045209a3a" gracePeriod=30
Oct 30 20:55:11.836673 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:11.836634    4471 generic.go:356] "Generic (PLEG): container finished" podID="ec075762-7e92-471c-bc35-c456ab936ad0" containerID="1f771057e49da49e2ccb9499f4089dbf7ffcb70fe9b050b74beaa33045209a3a" exitCode=0
Oct 30 20:55:11.836812 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:11.836724    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx" event={"ID":"ec075762-7e92-471c-bc35-c456ab936ad0","Type":"ContainerDied","Data":"1f771057e49da49e2ccb9499f4089dbf7ffcb70fe9b050b74beaa33045209a3a"}
Oct 30 20:55:11.871652 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:11.871619    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"
Oct 30 20:55:12.032756 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:12.032721    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec075762-7e92-471c-bc35-c456ab936ad0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec075762-7e92-471c-bc35-c456ab936ad0-openshift-service-ca\") pod \"ec075762-7e92-471c-bc35-c456ab936ad0\" (UID: \"ec075762-7e92-471c-bc35-c456ab936ad0\") "
Oct 30 20:55:12.032951 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:12.032781    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec075762-7e92-471c-bc35-c456ab936ad0-kube-api-access-dqxdq\" (UniqueName: \"kubernetes.io/projected/ec075762-7e92-471c-bc35-c456ab936ad0-kube-api-access-dqxdq\") pod \"ec075762-7e92-471c-bc35-c456ab936ad0\" (UID: \"ec075762-7e92-471c-bc35-c456ab936ad0\") "
Oct 30 20:55:12.032951 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:12.032906    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec075762-7e92-471c-bc35-c456ab936ad0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec075762-7e92-471c-bc35-c456ab936ad0-metrics-certs\") pod \"ec075762-7e92-471c-bc35-c456ab936ad0\" (UID: \"ec075762-7e92-471c-bc35-c456ab936ad0\") "
Oct 30 20:55:12.033084 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:12.033010    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec075762-7e92-471c-bc35-c456ab936ad0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec075762-7e92-471c-bc35-c456ab936ad0-maas-api-tls\") pod \"ec075762-7e92-471c-bc35-c456ab936ad0\" (UID: \"ec075762-7e92-471c-bc35-c456ab936ad0\") "
Oct 30 20:55:12.033230 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:12.033198    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec075762-7e92-471c-bc35-c456ab936ad0-openshift-service-ca" pod "ec075762-7e92-471c-bc35-c456ab936ad0" (UID: "ec075762-7e92-471c-bc35-c456ab936ad0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:12.036032 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:12.036000    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec075762-7e92-471c-bc35-c456ab936ad0-metrics-certs" pod "ec075762-7e92-471c-bc35-c456ab936ad0" (UID: "ec075762-7e92-471c-bc35-c456ab936ad0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:55:12.036165 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:12.036054    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec075762-7e92-471c-bc35-c456ab936ad0-maas-api-tls" pod "ec075762-7e92-471c-bc35-c456ab936ad0" (UID: "ec075762-7e92-471c-bc35-c456ab936ad0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:55:12.036229 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:12.036195    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec075762-7e92-471c-bc35-c456ab936ad0-kube-api-access-dqxdq" pod "ec075762-7e92-471c-bc35-c456ab936ad0" (UID: "ec075762-7e92-471c-bc35-c456ab936ad0"). InnerVolumeSpecName "kube-api-access-dqxdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:12.138963 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:12.136491    4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec075762-7e92-471c-bc35-c456ab936ad0-metrics-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:12.138963 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:12.136529    4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec075762-7e92-471c-bc35-c456ab936ad0-maas-api-tls\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:12.138963 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:12.136548    4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec075762-7e92-471c-bc35-c456ab936ad0-openshift-service-ca\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:12.138963 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:12.136563    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqxdq\" (UniqueName: \"kubernetes.io/projected/ec075762-7e92-471c-bc35-c456ab936ad0-kube-api-access-dqxdq\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:12.843717 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:12.843632    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"
Oct 30 20:55:12.843717 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:12.843662    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx" event={"ID":"ec075762-7e92-471c-bc35-c456ab936ad0","Type":"ContainerDied","Data":"22ad219649555dfb4a9a82a1eaa44094452ebcbf1b0d1974f7fa73aee6cf4043"}
Oct 30 20:55:12.844252 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:12.843734    4471 scope.go:122] "RemoveContainer" containerID="1f771057e49da49e2ccb9499f4089dbf7ffcb70fe9b050b74beaa33045209a3a"
Oct 30 20:55:13.008611 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:13.008580    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"]
Oct 30 20:55:13.058008 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:13.057976    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8ac60-7866fd4775-8xcnx"]
Oct 30 20:55:14.430870 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:14.430838    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec075762-7e92-471c-bc35-c456ab936ad0" path="/var/lib/kubelet/pods/ec075762-7e92-471c-bc35-c456ab936ad0/volumes"
Oct 30 20:55:15.229749 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:15.229709    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng" podUID="38d4bdac-79bc-48af-b3ac-4141651c0f86" containerName="payload-processing" containerID="cri-o://7d826776e5c0276cc5caa472c1bb7efce0c7b015c135d9a5f617cf4ff33a26e9" gracePeriod=25
Oct 30 20:55:15.485658 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:15.485601    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng"
Oct 30 20:55:15.593823 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:15.593795    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d4bdac-79bc-48af-b3ac-4141651c0f86-kube-api-access-qqx8k\" (UniqueName: \"kubernetes.io/projected/38d4bdac-79bc-48af-b3ac-4141651c0f86-kube-api-access-qqx8k\") pod \"38d4bdac-79bc-48af-b3ac-4141651c0f86\" (UID: \"38d4bdac-79bc-48af-b3ac-4141651c0f86\") "
Oct 30 20:55:15.594077 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:15.593924    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38d4bdac-79bc-48af-b3ac-4141651c0f86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d4bdac-79bc-48af-b3ac-4141651c0f86-plugins-config-volume\") pod \"38d4bdac-79bc-48af-b3ac-4141651c0f86\" (UID: \"38d4bdac-79bc-48af-b3ac-4141651c0f86\") "
Oct 30 20:55:15.594333 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:15.594301    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d4bdac-79bc-48af-b3ac-4141651c0f86-plugins-config-volume" pod "38d4bdac-79bc-48af-b3ac-4141651c0f86" (UID: "38d4bdac-79bc-48af-b3ac-4141651c0f86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:15.596087 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:15.596065    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d4bdac-79bc-48af-b3ac-4141651c0f86-kube-api-access-qqx8k" pod "38d4bdac-79bc-48af-b3ac-4141651c0f86" (UID: "38d4bdac-79bc-48af-b3ac-4141651c0f86"). InnerVolumeSpecName "kube-api-access-qqx8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:15.695373 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:15.695340    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqx8k\" (UniqueName: \"kubernetes.io/projected/38d4bdac-79bc-48af-b3ac-4141651c0f86-kube-api-access-qqx8k\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:15.695373 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:15.695373    4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d4bdac-79bc-48af-b3ac-4141651c0f86-plugins-config-volume\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:15.866847 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:15.866738    4471 generic.go:356] "Generic (PLEG): container finished" podID="38d4bdac-79bc-48af-b3ac-4141651c0f86" containerID="7d826776e5c0276cc5caa472c1bb7efce0c7b015c135d9a5f617cf4ff33a26e9" exitCode=0
Oct 30 20:55:15.866847 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:15.866793    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng" event={"ID":"38d4bdac-79bc-48af-b3ac-4141651c0f86","Type":"ContainerDied","Data":"7d826776e5c0276cc5caa472c1bb7efce0c7b015c135d9a5f617cf4ff33a26e9"}
Oct 30 20:55:15.866847 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:15.866826    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng"
Oct 30 20:55:15.867163 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:15.866869    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng" event={"ID":"38d4bdac-79bc-48af-b3ac-4141651c0f86","Type":"ContainerDied","Data":"bc6bdc0eb04d156112bcbb8496276e0f4cbbcca3818c5a385228d9894f3e65be"}
Oct 30 20:55:15.867163 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:15.866894    4471 scope.go:122] "RemoveContainer" containerID="7d826776e5c0276cc5caa472c1bb7efce0c7b015c135d9a5f617cf4ff33a26e9"
Oct 30 20:55:15.877127 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:15.877110    4471 scope.go:122] "RemoveContainer" containerID="7d826776e5c0276cc5caa472c1bb7efce0c7b015c135d9a5f617cf4ff33a26e9"
Oct 30 20:55:15.877418 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:55:15.877395    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d826776e5c0276cc5caa472c1bb7efce0c7b015c135d9a5f617cf4ff33a26e9\": container with ID starting with 7d826776e5c0276cc5caa472c1bb7efce0c7b015c135d9a5f617cf4ff33a26e9 not found: ID does not exist" containerID="7d826776e5c0276cc5caa472c1bb7efce0c7b015c135d9a5f617cf4ff33a26e9"
Oct 30 20:55:15.877525 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:15.877426    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d826776e5c0276cc5caa472c1bb7efce0c7b015c135d9a5f617cf4ff33a26e9"} err="failed to get container status \"7d826776e5c0276cc5caa472c1bb7efce0c7b015c135d9a5f617cf4ff33a26e9\": rpc error: code = NotFound desc = could not find container \"7d826776e5c0276cc5caa472c1bb7efce0c7b015c135d9a5f617cf4ff33a26e9\": container with ID starting with 7d826776e5c0276cc5caa472c1bb7efce0c7b015c135d9a5f617cf4ff33a26e9 not found: ID does not exist"
Oct 30 20:55:16.046816 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:16.046781    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng"]
Oct 30 20:55:16.052305 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:16.052283    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d71c23-6d4c77c49d-997ng"]
Oct 30 20:55:16.431271 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:16.431237    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d4bdac-79bc-48af-b3ac-4141651c0f86" path="/var/lib/kubelet/pods/38d4bdac-79bc-48af-b3ac-4141651c0f86/volumes"
Oct 30 20:55:22.135700 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.135665    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"]
Oct 30 20:55:22.136225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136093    4471 cpu_manager.go:414] "containerMap: removing container" podUID="2a7a1c0a-2a90-487e-9a94-780475a94fb5" containerName="payload-pre-processing"
Oct 30 20:55:22.136225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136105    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a7a1c0a-2a90-487e-9a94-780475a94fb5" containerName="payload-pre-processing"
Oct 30 20:55:22.136225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136123    4471 cpu_manager.go:414] "containerMap: removing container" podUID="d03eaf5f-735f-43cf-aa21-45901e9e6c04" containerName="istio-proxy"
Oct 30 20:55:22.136225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136129    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03eaf5f-735f-43cf-aa21-45901e9e6c04" containerName="istio-proxy"
Oct 30 20:55:22.136225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136138    4471 cpu_manager.go:414] "containerMap: removing container" podUID="4d739d8d-97fe-4c71-a9f4-d8d72ea853b6" containerName="payload-pre-processing"
Oct 30 20:55:22.136225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136143    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d739d8d-97fe-4c71-a9f4-d8d72ea853b6" containerName="payload-pre-processing"
Oct 30 20:55:22.136225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136152    4471 cpu_manager.go:414] "containerMap: removing container" podUID="e9dd5e0a-7d52-4c72-a274-348b97f5dcf1" containerName="payload-pre-processing"
Oct 30 20:55:22.136225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136156    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9dd5e0a-7d52-4c72-a274-348b97f5dcf1" containerName="payload-pre-processing"
Oct 30 20:55:22.136225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136164    4471 cpu_manager.go:414] "containerMap: removing container" podUID="19dfa0fc-a01f-495b-8d9f-e6e976bef433" containerName="istio-proxy"
Oct 30 20:55:22.136225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136169    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19dfa0fc-a01f-495b-8d9f-e6e976bef433" containerName="istio-proxy"
Oct 30 20:55:22.136225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136176    4471 cpu_manager.go:414] "containerMap: removing container" podUID="38d4bdac-79bc-48af-b3ac-4141651c0f86" containerName="payload-processing"
Oct 30 20:55:22.136225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136182    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d4bdac-79bc-48af-b3ac-4141651c0f86" containerName="payload-processing"
Oct 30 20:55:22.136225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136188    4471 cpu_manager.go:414] "containerMap: removing container" podUID="3613a886-6b08-4e25-807a-fbbe5c080775" containerName="istio-proxy"
Oct 30 20:55:22.136225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136192    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3613a886-6b08-4e25-807a-fbbe5c080775" containerName="istio-proxy"
Oct 30 20:55:22.136225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136200    4471 cpu_manager.go:414] "containerMap: removing container" podUID="0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9" containerName="maas-api"
Oct 30 20:55:22.136225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136204    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9" containerName="maas-api"
Oct 30 20:55:22.136225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136212    4471 cpu_manager.go:414] "containerMap: removing container" podUID="c7b255dd-817d-480d-9f58-df362bca182d" containerName="istio-proxy"
Oct 30 20:55:22.136225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136217    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b255dd-817d-480d-9f58-df362bca182d" containerName="istio-proxy"
Oct 30 20:55:22.136225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136225    4471 cpu_manager.go:414] "containerMap: removing container" podUID="b2d2f36f-2a1b-402b-8a71-08f174e078c4" containerName="istio-proxy"
Oct 30 20:55:22.136225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136229    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2d2f36f-2a1b-402b-8a71-08f174e078c4" containerName="istio-proxy"
Oct 30 20:55:22.136225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136235    4471 cpu_manager.go:414] "containerMap: removing container" podUID="ec075762-7e92-471c-bc35-c456ab936ad0" containerName="maas-api"
Oct 30 20:55:22.136225 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136239    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec075762-7e92-471c-bc35-c456ab936ad0" containerName="maas-api"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136248    4471 cpu_manager.go:414] "containerMap: removing container" podUID="e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12" containerName="payload-pre-processing"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136253    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12" containerName="payload-pre-processing"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136258    4471 cpu_manager.go:414] "containerMap: removing container" podUID="a2199190-0e92-4238-8011-84a7d273a53e" containerName="maas-api"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136263    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2199190-0e92-4238-8011-84a7d273a53e" containerName="maas-api"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136270    4471 cpu_manager.go:414] "containerMap: removing container" podUID="7b0db74e-64b2-4bb2-83da-ee1669156580" containerName="istio-proxy"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136274    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b0db74e-64b2-4bb2-83da-ee1669156580" containerName="istio-proxy"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136281    4471 cpu_manager.go:414] "containerMap: removing container" podUID="c121a634-e346-4be2-872d-fc943d82ad37" containerName="payload-processing"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136285    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c121a634-e346-4be2-872d-fc943d82ad37" containerName="payload-processing"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136291    4471 cpu_manager.go:414] "containerMap: removing container" podUID="f1bf6069-9c41-4f32-83c6-91561d7833b4" containerName="payload-processing"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136295    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1bf6069-9c41-4f32-83c6-91561d7833b4" containerName="payload-processing"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136300    4471 cpu_manager.go:414] "containerMap: removing container" podUID="84057aab-629f-4918-9d85-cd3e2d3a95b4" containerName="payload-pre-processing"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136304    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84057aab-629f-4918-9d85-cd3e2d3a95b4" containerName="payload-pre-processing"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136310    4471 cpu_manager.go:414] "containerMap: removing container" podUID="165b506b-c809-43eb-a183-686869cf35db" containerName="istio-proxy"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136314    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165b506b-c809-43eb-a183-686869cf35db" containerName="istio-proxy"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136340    4471 cpu_manager.go:414] "containerMap: removing container" podUID="03e1b048-5b3f-43fd-b321-897f8d6e6e53" containerName="istio-proxy"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136345    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e1b048-5b3f-43fd-b321-897f8d6e6e53" containerName="istio-proxy"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136353    4471 cpu_manager.go:414] "containerMap: removing container" podUID="67597528-00c9-4a1d-a5ed-02a170390a40" containerName="payload-processing"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136357    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67597528-00c9-4a1d-a5ed-02a170390a40" containerName="payload-processing"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136439    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec075762-7e92-471c-bc35-c456ab936ad0" containerName="maas-api"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136446    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e339ad-9117-4d8d-a11e-c9d2fd7ebb12" containerName="payload-pre-processing"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136452    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2199190-0e92-4238-8011-84a7d273a53e" containerName="maas-api"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136459    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b0db74e-64b2-4bb2-83da-ee1669156580" containerName="istio-proxy"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136465    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="c121a634-e346-4be2-872d-fc943d82ad37" containerName="payload-processing"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136471    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1bf6069-9c41-4f32-83c6-91561d7833b4" containerName="payload-processing"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136477    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="84057aab-629f-4918-9d85-cd3e2d3a95b4" containerName="payload-pre-processing"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136483    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="165b506b-c809-43eb-a183-686869cf35db" containerName="istio-proxy"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136490    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e1b048-5b3f-43fd-b321-897f8d6e6e53" containerName="istio-proxy"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136495    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="67597528-00c9-4a1d-a5ed-02a170390a40" containerName="payload-processing"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136500    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b255dd-817d-480d-9f58-df362bca182d" containerName="istio-proxy"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136505    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a7a1c0a-2a90-487e-9a94-780475a94fb5" containerName="payload-pre-processing"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136511    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03eaf5f-735f-43cf-aa21-45901e9e6c04" containerName="istio-proxy"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136516    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d739d8d-97fe-4c71-a9f4-d8d72ea853b6" containerName="payload-pre-processing"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136522    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9dd5e0a-7d52-4c72-a274-348b97f5dcf1" containerName="payload-pre-processing"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136528    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="19dfa0fc-a01f-495b-8d9f-e6e976bef433" containerName="istio-proxy"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136533    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d4bdac-79bc-48af-b3ac-4141651c0f86" containerName="payload-processing"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136537    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="3613a886-6b08-4e25-807a-fbbe5c080775" containerName="istio-proxy"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136542    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ad51c43-2b3f-4b2e-bc30-5e847bcad9f9" containerName="maas-api"
Oct 30 20:55:22.137398 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.136549    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2d2f36f-2a1b-402b-8a71-08f174e078c4" containerName="istio-proxy"
Oct 30 20:55:22.141101 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.141078    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.168284 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.168248    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-podinfo\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.168856 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.168816    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-envoy\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.169033 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.169014    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/675355ae-1aae-4e8e-b12f-d3727346ae51-istiod-ca-cert\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.169178 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.169163    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-token\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.169293 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.169278    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmw2j\" (UniqueName: \"kubernetes.io/projected/675355ae-1aae-4e8e-b12f-d3727346ae51-kube-api-access-rmw2j\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.169494 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.169475    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-credential-socket\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.169615 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.169601    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-workload-socket\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.169781 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.169766    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-workload-certs\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.169974 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.169957    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-data\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.197460 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.197411    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5dc13e-openshift-default-dockercfg-wkpbn\""
Oct 30 20:55:22.275026 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.274980    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-credential-socket\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.275301 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.275276    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-workload-socket\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.275586 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.275564    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-workload-certs\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.275746 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.275730    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-data\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.285355 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.278436    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-podinfo\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.285355 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.278522    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-envoy\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.285355 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.278550    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/675355ae-1aae-4e8e-b12f-d3727346ae51-istiod-ca-cert\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.285355 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.278607    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-token\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.285355 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.278634    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmw2j\" (UniqueName: \"kubernetes.io/projected/675355ae-1aae-4e8e-b12f-d3727346ae51-kube-api-access-rmw2j\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.285355 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.279491    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-credential-socket\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.285355 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.279730    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-workload-socket\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.285355 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.279956    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-workload-certs\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.285355 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.280248    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-data\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.303460 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.303415    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"]
Oct 30 20:55:22.341040 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.304571    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/675355ae-1aae-4e8e-b12f-d3727346ae51-istiod-ca-cert\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.363348 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.361770    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-envoy\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.363348 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.363159    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-podinfo\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.378890 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.377288    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmw2j\" (UniqueName: \"kubernetes.io/projected/675355ae-1aae-4e8e-b12f-d3727346ae51-kube-api-access-rmw2j\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.426423 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.426345    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-token\") pod \"e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") " pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.459431 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.456827    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:22.907156 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.907118    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"]
Oct 30 20:55:22.909175 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:55:22.909142    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675355ae_1aae_4e8e_b12f_d3727346ae51.slice/crio-ff0e1307b3496b07156caa182902dcd27ec75b8b4f6299f3852fb18dce23f00c WatchSource:0}: Error finding container ff0e1307b3496b07156caa182902dcd27ec75b8b4f6299f3852fb18dce23f00c: Status 404 returned error can't find the container with id ff0e1307b3496b07156caa182902dcd27ec75b8b4f6299f3852fb18dce23f00c
Oct 30 20:55:22.912068 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.912032    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:55:22.912161 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.912110    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:55:22.912161 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.912139    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:55:22.920007 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:22.919981    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb" event={"ID":"675355ae-1aae-4e8e-b12f-d3727346ae51","Type":"ContainerStarted","Data":"ff0e1307b3496b07156caa182902dcd27ec75b8b4f6299f3852fb18dce23f00c"}
Oct 30 20:55:23.926904 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:23.926872    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb" event={"ID":"675355ae-1aae-4e8e-b12f-d3727346ae51","Type":"ContainerStarted","Data":"a885dbf30b99c4d0684478872187a3679004c6545cdc93abe6ce5d4e69f9cecf"}
Oct 30 20:55:24.143955 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:24.143898    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb" podStartSLOduration=2.143885126 podStartE2EDuration="2.143885126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:24.093300922 +0000 UTC m=+2143.771929538" watchObservedRunningTime="2026-10-30 20:55:24.143885126 +0000 UTC m=+2143.822513742"
Oct 30 20:55:24.458271 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:24.458232    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:24.465806 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:24.465778    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:24.931778 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:24.931747    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:24.932611 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:24.932593    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:55:27.662802 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:27.662768    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"]
Oct 30 20:55:27.663230 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:27.662954    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp" podUID="3c6e1eb7-98b5-436a-b84f-a2f4528904dc" containerName="maas-api" containerID="cri-o://4045f3371a60a44990d0c603a6e0d5e39e6af1e83670e2240799a7c4b30c317e" gracePeriod=30
Oct 30 20:55:27.953530 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:27.953495    4471 generic.go:356] "Generic (PLEG): container finished" podID="3c6e1eb7-98b5-436a-b84f-a2f4528904dc" containerID="4045f3371a60a44990d0c603a6e0d5e39e6af1e83670e2240799a7c4b30c317e" exitCode=0
Oct 30 20:55:27.953703 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:27.953524    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp" event={"ID":"3c6e1eb7-98b5-436a-b84f-a2f4528904dc","Type":"ContainerDied","Data":"4045f3371a60a44990d0c603a6e0d5e39e6af1e83670e2240799a7c4b30c317e"}
Oct 30 20:55:27.980057 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:27.980025    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"
Oct 30 20:55:28.076432 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:28.076385    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-maas-api-tls\") pod \"3c6e1eb7-98b5-436a-b84f-a2f4528904dc\" (UID: \"3c6e1eb7-98b5-436a-b84f-a2f4528904dc\") "
Oct 30 20:55:28.076635 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:28.076456    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-kube-api-access-cnz64\" (UniqueName: \"kubernetes.io/projected/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-kube-api-access-cnz64\") pod \"3c6e1eb7-98b5-436a-b84f-a2f4528904dc\" (UID: \"3c6e1eb7-98b5-436a-b84f-a2f4528904dc\") "
Oct 30 20:55:28.076635 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:28.076500    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-metrics-certs\") pod \"3c6e1eb7-98b5-436a-b84f-a2f4528904dc\" (UID: \"3c6e1eb7-98b5-436a-b84f-a2f4528904dc\") "
Oct 30 20:55:28.076635 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:28.076594    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-openshift-service-ca\") pod \"3c6e1eb7-98b5-436a-b84f-a2f4528904dc\" (UID: \"3c6e1eb7-98b5-436a-b84f-a2f4528904dc\") "
Oct 30 20:55:28.077035 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:28.077009    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-openshift-service-ca" pod "3c6e1eb7-98b5-436a-b84f-a2f4528904dc" (UID: "3c6e1eb7-98b5-436a-b84f-a2f4528904dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:28.077130 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:28.077110    4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-openshift-service-ca\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:28.078865 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:28.078834    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-kube-api-access-cnz64" pod "3c6e1eb7-98b5-436a-b84f-a2f4528904dc" (UID: "3c6e1eb7-98b5-436a-b84f-a2f4528904dc"). InnerVolumeSpecName "kube-api-access-cnz64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:28.078991 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:28.078933    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-metrics-certs" pod "3c6e1eb7-98b5-436a-b84f-a2f4528904dc" (UID: "3c6e1eb7-98b5-436a-b84f-a2f4528904dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:55:28.079063 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:28.079045    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-maas-api-tls" pod "3c6e1eb7-98b5-436a-b84f-a2f4528904dc" (UID: "3c6e1eb7-98b5-436a-b84f-a2f4528904dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:55:28.178457 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:28.178413    4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-maas-api-tls\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:28.178457 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:28.178448    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnz64\" (UniqueName: \"kubernetes.io/projected/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-kube-api-access-cnz64\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:28.178457 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:28.178459    4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6e1eb7-98b5-436a-b84f-a2f4528904dc-metrics-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:28.959763 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:28.959676    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"
Oct 30 20:55:28.959763 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:28.959709    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp" event={"ID":"3c6e1eb7-98b5-436a-b84f-a2f4528904dc","Type":"ContainerDied","Data":"e585fe4c65e0629236a6648e59f31245cf66f0e5eb38798781184f3bb33ce293"}
Oct 30 20:55:28.960266 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:28.959779    4471 scope.go:122] "RemoveContainer" containerID="4045f3371a60a44990d0c603a6e0d5e39e6af1e83670e2240799a7c4b30c317e"
Oct 30 20:55:29.123640 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:29.123604    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"]
Oct 30 20:55:29.171254 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:29.171219    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d71c23-694d9758c-k55gp"]
Oct 30 20:55:30.452124 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:30.452061    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c6e1eb7-98b5-436a-b84f-a2f4528904dc" path="/var/lib/kubelet/pods/3c6e1eb7-98b5-436a-b84f-a2f4528904dc/volumes"
Oct 30 20:55:30.915732 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:30.915694    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz"]
Oct 30 20:55:30.916668 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:30.916639    4471 cpu_manager.go:414] "containerMap: removing container" podUID="c7b255dd-817d-480d-9f58-df362bca182d" containerName="istio-proxy"
Oct 30 20:55:30.916826 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:30.916810    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b255dd-817d-480d-9f58-df362bca182d" containerName="istio-proxy"
Oct 30 20:55:30.916958 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:30.916946    4471 cpu_manager.go:414] "containerMap: removing container" podUID="3c6e1eb7-98b5-436a-b84f-a2f4528904dc" containerName="maas-api"
Oct 30 20:55:30.917053 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:30.917040    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c6e1eb7-98b5-436a-b84f-a2f4528904dc" containerName="maas-api"
Oct 30 20:55:30.917310 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:30.917294    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b255dd-817d-480d-9f58-df362bca182d" containerName="istio-proxy"
Oct 30 20:55:30.917445 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:30.917427    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c6e1eb7-98b5-436a-b84f-a2f4528904dc" containerName="maas-api"
Oct 30 20:55:30.923952 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:30.923424    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz"
Oct 30 20:55:30.965662 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:55:30.965623    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:55:30.981592 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:30.981556    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz"]
Oct 30 20:55:31.002453 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.002427    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5dc13e\""
Oct 30 20:55:31.002999 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.002976    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5dc13e-dockercfg-njczp\""
Oct 30 20:55:31.030040 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.030014    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff6334a-529c-411f-ae61-db2d2834906d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz\" (UID: \"aff6334a-529c-411f-ae61-db2d2834906d\") " pod="openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz"
Oct 30 20:55:31.030174 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.030085    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qlgf\" (UniqueName: \"kubernetes.io/projected/aff6334a-529c-411f-ae61-db2d2834906d-kube-api-access-8qlgf\") pod \"payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz\" (UID: \"aff6334a-529c-411f-ae61-db2d2834906d\") " pod="openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz"
Oct 30 20:55:31.089405 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.089370    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf"]
Oct 30 20:55:31.093097 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.093076    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf"
Oct 30 20:55:31.131387 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.131360    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff6334a-529c-411f-ae61-db2d2834906d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz\" (UID: \"aff6334a-529c-411f-ae61-db2d2834906d\") " pod="openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz"
Oct 30 20:55:31.131493 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.131441    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qlgf\" (UniqueName: \"kubernetes.io/projected/aff6334a-529c-411f-ae61-db2d2834906d-kube-api-access-8qlgf\") pod \"payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz\" (UID: \"aff6334a-529c-411f-ae61-db2d2834906d\") " pod="openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz"
Oct 30 20:55:31.132059 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.132038    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff6334a-529c-411f-ae61-db2d2834906d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz\" (UID: \"aff6334a-529c-411f-ae61-db2d2834906d\") " pod="openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz"
Oct 30 20:55:31.160135 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.160109    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf"]
Oct 30 20:55:31.192060 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.191989    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qlgf\" (UniqueName: \"kubernetes.io/projected/aff6334a-529c-411f-ae61-db2d2834906d-kube-api-access-8qlgf\") pod \"payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz\" (UID: \"aff6334a-529c-411f-ae61-db2d2834906d\") " pod="openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz"
Oct 30 20:55:31.232435 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.232403    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6kkf\" (UniqueName: \"kubernetes.io/projected/39317fa6-714f-4826-9a67-f16b11f3f7e7-kube-api-access-s6kkf\") pod \"payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf\" (UID: \"39317fa6-714f-4826-9a67-f16b11f3f7e7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf"
Oct 30 20:55:31.232578 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.232443    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39317fa6-714f-4826-9a67-f16b11f3f7e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf\" (UID: \"39317fa6-714f-4826-9a67-f16b11f3f7e7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf"
Oct 30 20:55:31.241994 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.241965    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz"
Oct 30 20:55:31.333918 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.333876    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6kkf\" (UniqueName: \"kubernetes.io/projected/39317fa6-714f-4826-9a67-f16b11f3f7e7-kube-api-access-s6kkf\") pod \"payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf\" (UID: \"39317fa6-714f-4826-9a67-f16b11f3f7e7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf"
Oct 30 20:55:31.333918 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.333916    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39317fa6-714f-4826-9a67-f16b11f3f7e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf\" (UID: \"39317fa6-714f-4826-9a67-f16b11f3f7e7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf"
Oct 30 20:55:31.334927 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.334899    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39317fa6-714f-4826-9a67-f16b11f3f7e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf\" (UID: \"39317fa6-714f-4826-9a67-f16b11f3f7e7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf"
Oct 30 20:55:31.391181 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.391152    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6kkf\" (UniqueName: \"kubernetes.io/projected/39317fa6-714f-4826-9a67-f16b11f3f7e7-kube-api-access-s6kkf\") pod \"payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf\" (UID: \"39317fa6-714f-4826-9a67-f16b11f3f7e7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf"
Oct 30 20:55:31.406038 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.406013    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf"
Oct 30 20:55:31.617900 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.617866    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz"]
Oct 30 20:55:31.621389 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:55:31.621357    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff6334a_529c_411f_ae61_db2d2834906d.slice/crio-851b78b329b4b64dc7524efe396bb4cbd0483a530c26f03fe3661d46138fc98c WatchSource:0}: Error finding container 851b78b329b4b64dc7524efe396bb4cbd0483a530c26f03fe3661d46138fc98c: Status 404 returned error can't find the container with id 851b78b329b4b64dc7524efe396bb4cbd0483a530c26f03fe3661d46138fc98c
Oct 30 20:55:31.782765 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.782740    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf"]
Oct 30 20:55:31.784704 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:55:31.784678    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39317fa6_714f_4826_9a67_f16b11f3f7e7.slice/crio-6b2481c000e4768a54cdd601a854387c18a523c1a61570895710646a10bf092d WatchSource:0}: Error finding container 6b2481c000e4768a54cdd601a854387c18a523c1a61570895710646a10bf092d: Status 404 returned error can't find the container with id 6b2481c000e4768a54cdd601a854387c18a523c1a61570895710646a10bf092d
Oct 30 20:55:31.978745 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.978712    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf" event={"ID":"39317fa6-714f-4826-9a67-f16b11f3f7e7","Type":"ContainerStarted","Data":"8f221c4a41765a8dfb4ec6552a8f1a9b63094f2d8c83fda2dd4de69b08c3535c"}
Oct 30 20:55:31.978918 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.978759    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf" event={"ID":"39317fa6-714f-4826-9a67-f16b11f3f7e7","Type":"ContainerStarted","Data":"6b2481c000e4768a54cdd601a854387c18a523c1a61570895710646a10bf092d"}
Oct 30 20:55:31.978918 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.978877    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf"
Oct 30 20:55:31.980606 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.980587    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz" event={"ID":"aff6334a-529c-411f-ae61-db2d2834906d","Type":"ContainerStarted","Data":"f4cd4ae522050e810882c0b2250e46f70e49e2344ae4c3b5634a43dacf91dd67"}
Oct 30 20:55:31.980692 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.980631    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz"
Oct 30 20:55:31.980692 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:31.980649    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz" event={"ID":"aff6334a-529c-411f-ae61-db2d2834906d","Type":"ContainerStarted","Data":"851b78b329b4b64dc7524efe396bb4cbd0483a530c26f03fe3661d46138fc98c"}
Oct 30 20:55:32.093342 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:32.093220    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf" podStartSLOduration=1.093205859 podStartE2EDuration="1.093205859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:32.091951844 +0000 UTC m=+2151.770580462" watchObservedRunningTime="2026-10-30 20:55:32.093205859 +0000 UTC m=+2151.771834475"
Oct 30 20:55:32.219031 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:32.216022    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz" podStartSLOduration=2.216004711 podStartE2EDuration="2.216004711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:32.215670056 +0000 UTC m=+2151.894298673" watchObservedRunningTime="2026-10-30 20:55:32.216004711 +0000 UTC m=+2151.894633328"
Oct 30 20:55:42.989465 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:42.989436    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz"
Oct 30 20:55:42.989894 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:42.989500    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf"
Oct 30 20:55:50.011252 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:50.011217    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"]
Oct 30 20:55:50.016278 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:50.016259    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"
Oct 30 20:55:50.121343 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:50.119453    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-bcef0e-serving-cert\""
Oct 30 20:55:50.123431 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:50.123406    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"]
Oct 30 20:55:50.146613 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:50.146575    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4mjr\" (UniqueName: \"kubernetes.io/projected/811787ae-bcf5-452e-9c13-2ada1d001705-kube-api-access-t4mjr\") pod \"maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv\" (UID: \"811787ae-bcf5-452e-9c13-2ada1d001705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"
Oct 30 20:55:50.146754 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:50.146740    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/811787ae-bcf5-452e-9c13-2ada1d001705-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv\" (UID: \"811787ae-bcf5-452e-9c13-2ada1d001705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"
Oct 30 20:55:50.146825 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:50.146783    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/811787ae-bcf5-452e-9c13-2ada1d001705-metrics-certs\") pod \"maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv\" (UID: \"811787ae-bcf5-452e-9c13-2ada1d001705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"
Oct 30 20:55:50.146891 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:50.146879    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/811787ae-bcf5-452e-9c13-2ada1d001705-maas-api-tls\") pod \"maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv\" (UID: \"811787ae-bcf5-452e-9c13-2ada1d001705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"
Oct 30 20:55:50.247831 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:50.247803    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/811787ae-bcf5-452e-9c13-2ada1d001705-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv\" (UID: \"811787ae-bcf5-452e-9c13-2ada1d001705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"
Oct 30 20:55:50.248013 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:50.247855    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/811787ae-bcf5-452e-9c13-2ada1d001705-metrics-certs\") pod \"maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv\" (UID: \"811787ae-bcf5-452e-9c13-2ada1d001705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"
Oct 30 20:55:50.248013 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:50.247901    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/811787ae-bcf5-452e-9c13-2ada1d001705-maas-api-tls\") pod \"maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv\" (UID: \"811787ae-bcf5-452e-9c13-2ada1d001705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"
Oct 30 20:55:50.248013 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:50.247932    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4mjr\" (UniqueName: \"kubernetes.io/projected/811787ae-bcf5-452e-9c13-2ada1d001705-kube-api-access-t4mjr\") pod \"maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv\" (UID: \"811787ae-bcf5-452e-9c13-2ada1d001705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"
Oct 30 20:55:50.248172 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:55:50.248040    4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-serving-cert: secret "maas-api-e2e-inf-a-bcef0e-serving-cert" not found
Oct 30 20:55:50.248172 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:55:50.248158    4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/811787ae-bcf5-452e-9c13-2ada1d001705-maas-api-tls podName:811787ae-bcf5-452e-9c13-2ada1d001705 nodeName:}" failed. No retries permitted until 2026-10-30 20:55:50.748133865 +0000 UTC m=+2170.426762498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/811787ae-bcf5-452e-9c13-2ada1d001705-maas-api-tls") pod "maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv" (UID: "811787ae-bcf5-452e-9c13-2ada1d001705") : secret "maas-api-e2e-inf-a-bcef0e-serving-cert" not found
Oct 30 20:55:50.248623 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:50.248600    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/811787ae-bcf5-452e-9c13-2ada1d001705-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv\" (UID: \"811787ae-bcf5-452e-9c13-2ada1d001705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"
Oct 30 20:55:50.250512 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:50.250492    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/811787ae-bcf5-452e-9c13-2ada1d001705-metrics-certs\") pod \"maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv\" (UID: \"811787ae-bcf5-452e-9c13-2ada1d001705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"
Oct 30 20:55:50.307948 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:50.307845    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4mjr\" (UniqueName: \"kubernetes.io/projected/811787ae-bcf5-452e-9c13-2ada1d001705-kube-api-access-t4mjr\") pod \"maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv\" (UID: \"811787ae-bcf5-452e-9c13-2ada1d001705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"
Oct 30 20:55:50.754693 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:50.754653    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/811787ae-bcf5-452e-9c13-2ada1d001705-maas-api-tls\") pod \"maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv\" (UID: \"811787ae-bcf5-452e-9c13-2ada1d001705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"
Oct 30 20:55:50.757169 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:50.757143    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/811787ae-bcf5-452e-9c13-2ada1d001705-maas-api-tls\") pod \"maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv\" (UID: \"811787ae-bcf5-452e-9c13-2ada1d001705\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"
Oct 30 20:55:50.929505 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:50.929466    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"
Oct 30 20:55:51.307769 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:51.307743    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"]
Oct 30 20:55:51.309988 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:55:51.309955    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811787ae_bcf5_452e_9c13_2ada1d001705.slice/crio-672939958a032a82e236e03ff11f80c6b2f2c637bb25d1869c17e34292d2d471 WatchSource:0}: Error finding container 672939958a032a82e236e03ff11f80c6b2f2c637bb25d1869c17e34292d2d471: Status 404 returned error can't find the container with id 672939958a032a82e236e03ff11f80c6b2f2c637bb25d1869c17e34292d2d471
Oct 30 20:55:52.120179 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:52.120132    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv" event={"ID":"811787ae-bcf5-452e-9c13-2ada1d001705","Type":"ContainerStarted","Data":"672939958a032a82e236e03ff11f80c6b2f2c637bb25d1869c17e34292d2d471"}
Oct 30 20:55:52.252798 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:52.252770    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt"]
Oct 30 20:55:53.126990 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:53.126952    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv" event={"ID":"811787ae-bcf5-452e-9c13-2ada1d001705","Type":"ContainerStarted","Data":"042b697cc31f6527ba5389c602f49385ccb73ee3d45c276c1633c8e807c43c6b"}
Oct 30 20:55:53.127421 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:53.127039    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"
Oct 30 20:55:57.253114 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:57.253063    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt" podUID="93409cb5-1925-4012-8983-1491ead93a97" containerName="payload-pre-processing" containerID="cri-o://1e470e5f528c8b77f4bdea4916d5a7d164423ab5f996cf73465e04500078793c" gracePeriod=25
Oct 30 20:55:57.516699 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:57.516635    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt"
Oct 30 20:55:57.630534 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:57.630475    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv" podStartSLOduration=7.709715305 podStartE2EDuration="8.630459936s" totalImagesPullingTime="920.744631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:53.250830309 +0000 UTC m=+2172.929458927" watchObservedRunningTime="2026-10-30 20:55:57.630459936 +0000 UTC m=+2177.309088555"
Oct 30 20:55:57.685929 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:57.685894    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93409cb5-1925-4012-8983-1491ead93a97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93409cb5-1925-4012-8983-1491ead93a97-plugins-config-volume\") pod \"93409cb5-1925-4012-8983-1491ead93a97\" (UID: \"93409cb5-1925-4012-8983-1491ead93a97\") "
Oct 30 20:55:57.686091 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:57.686064    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93409cb5-1925-4012-8983-1491ead93a97-kube-api-access-k8twz\" (UniqueName: \"kubernetes.io/projected/93409cb5-1925-4012-8983-1491ead93a97-kube-api-access-k8twz\") pod \"93409cb5-1925-4012-8983-1491ead93a97\" (UID: \"93409cb5-1925-4012-8983-1491ead93a97\") "
Oct 30 20:55:57.686295 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:57.686272    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93409cb5-1925-4012-8983-1491ead93a97-plugins-config-volume" pod "93409cb5-1925-4012-8983-1491ead93a97" (UID: "93409cb5-1925-4012-8983-1491ead93a97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:57.688280 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:57.688250    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93409cb5-1925-4012-8983-1491ead93a97-kube-api-access-k8twz" pod "93409cb5-1925-4012-8983-1491ead93a97" (UID: "93409cb5-1925-4012-8983-1491ead93a97"). InnerVolumeSpecName "kube-api-access-k8twz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:57.787353 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:57.787244    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8twz\" (UniqueName: \"kubernetes.io/projected/93409cb5-1925-4012-8983-1491ead93a97-kube-api-access-k8twz\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:57.787353 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:57.787275    4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93409cb5-1925-4012-8983-1491ead93a97-plugins-config-volume\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:58.156261 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:58.156230    4471 generic.go:356] "Generic (PLEG): container finished" podID="93409cb5-1925-4012-8983-1491ead93a97" containerID="1e470e5f528c8b77f4bdea4916d5a7d164423ab5f996cf73465e04500078793c" exitCode=0
Oct 30 20:55:58.156456 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:58.156279    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt" event={"ID":"93409cb5-1925-4012-8983-1491ead93a97","Type":"ContainerDied","Data":"1e470e5f528c8b77f4bdea4916d5a7d164423ab5f996cf73465e04500078793c"}
Oct 30 20:55:58.156456 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:58.156282    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt"
Oct 30 20:55:58.156456 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:58.156359    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt" event={"ID":"93409cb5-1925-4012-8983-1491ead93a97","Type":"ContainerDied","Data":"989fa666df8dd1b86f2a2345eb10f16971a44cb307e9c7fbda9ddd82dfea840a"}
Oct 30 20:55:58.156456 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:58.156382    4471 scope.go:122] "RemoveContainer" containerID="1e470e5f528c8b77f4bdea4916d5a7d164423ab5f996cf73465e04500078793c"
Oct 30 20:55:58.166641 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:58.166622    4471 scope.go:122] "RemoveContainer" containerID="1e470e5f528c8b77f4bdea4916d5a7d164423ab5f996cf73465e04500078793c"
Oct 30 20:55:58.166914 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:55:58.166895    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e470e5f528c8b77f4bdea4916d5a7d164423ab5f996cf73465e04500078793c\": container with ID starting with 1e470e5f528c8b77f4bdea4916d5a7d164423ab5f996cf73465e04500078793c not found: ID does not exist" containerID="1e470e5f528c8b77f4bdea4916d5a7d164423ab5f996cf73465e04500078793c"
Oct 30 20:55:58.166966 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:58.166922    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e470e5f528c8b77f4bdea4916d5a7d164423ab5f996cf73465e04500078793c"} err="failed to get container status \"1e470e5f528c8b77f4bdea4916d5a7d164423ab5f996cf73465e04500078793c\": rpc error: code = NotFound desc = could not find container \"1e470e5f528c8b77f4bdea4916d5a7d164423ab5f996cf73465e04500078793c\": container with ID starting with 1e470e5f528c8b77f4bdea4916d5a7d164423ab5f996cf73465e04500078793c not found: ID does not exist"
Oct 30 20:55:58.320140 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:58.320102    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt"]
Oct 30 20:55:58.368841 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:58.368810    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-061bda-76bd5fbf9f-q6stt"]
Oct 30 20:55:58.431583 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:58.431507    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93409cb5-1925-4012-8983-1491ead93a97" path="/var/lib/kubelet/pods/93409cb5-1925-4012-8983-1491ead93a97/volumes"
Oct 30 20:55:59.138637 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:55:59.138609    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"
Oct 30 20:56:00.954603 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:56:00.954572    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:56:04.875868 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:04.875831    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"]
Oct 30 20:56:04.876308 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:04.876049    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8" podUID="4c185fa5-18ff-4caf-b853-dee1d4b0432f" containerName="maas-api" containerID="cri-o://14479cbd6454ed9359d134bcb08dd938abdebf6003de65e2a8d4658d99a29896" gracePeriod=30
Oct 30 20:56:05.169208 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:05.167758    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"
Oct 30 20:56:05.209457 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:05.209413    4471 generic.go:356] "Generic (PLEG): container finished" podID="4c185fa5-18ff-4caf-b853-dee1d4b0432f" containerID="14479cbd6454ed9359d134bcb08dd938abdebf6003de65e2a8d4658d99a29896" exitCode=0
Oct 30 20:56:05.209656 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:05.209505    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8" event={"ID":"4c185fa5-18ff-4caf-b853-dee1d4b0432f","Type":"ContainerDied","Data":"14479cbd6454ed9359d134bcb08dd938abdebf6003de65e2a8d4658d99a29896"}
Oct 30 20:56:05.209656 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:05.209560    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8" event={"ID":"4c185fa5-18ff-4caf-b853-dee1d4b0432f","Type":"ContainerDied","Data":"c5f0822008d80059e56a4a9908fb52c71632f41eea6bc1e4813a3848e7476210"}
Oct 30 20:56:05.209656 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:05.209606    4471 scope.go:122] "RemoveContainer" containerID="14479cbd6454ed9359d134bcb08dd938abdebf6003de65e2a8d4658d99a29896"
Oct 30 20:56:05.209836 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:05.209658    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"
Oct 30 20:56:05.220032 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:05.220014    4471 scope.go:122] "RemoveContainer" containerID="14479cbd6454ed9359d134bcb08dd938abdebf6003de65e2a8d4658d99a29896"
Oct 30 20:56:05.220357 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:56:05.220306    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14479cbd6454ed9359d134bcb08dd938abdebf6003de65e2a8d4658d99a29896\": container with ID starting with 14479cbd6454ed9359d134bcb08dd938abdebf6003de65e2a8d4658d99a29896 not found: ID does not exist" containerID="14479cbd6454ed9359d134bcb08dd938abdebf6003de65e2a8d4658d99a29896"
Oct 30 20:56:05.220433 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:05.220368    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14479cbd6454ed9359d134bcb08dd938abdebf6003de65e2a8d4658d99a29896"} err="failed to get container status \"14479cbd6454ed9359d134bcb08dd938abdebf6003de65e2a8d4658d99a29896\": rpc error: code = NotFound desc = could not find container \"14479cbd6454ed9359d134bcb08dd938abdebf6003de65e2a8d4658d99a29896\": container with ID starting with 14479cbd6454ed9359d134bcb08dd938abdebf6003de65e2a8d4658d99a29896 not found: ID does not exist"
Oct 30 20:56:05.267671 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:05.267643    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c185fa5-18ff-4caf-b853-dee1d4b0432f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c185fa5-18ff-4caf-b853-dee1d4b0432f-metrics-certs\") pod \"4c185fa5-18ff-4caf-b853-dee1d4b0432f\" (UID: \"4c185fa5-18ff-4caf-b853-dee1d4b0432f\") "
Oct 30 20:56:05.267816 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:05.267793    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c185fa5-18ff-4caf-b853-dee1d4b0432f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c185fa5-18ff-4caf-b853-dee1d4b0432f-maas-api-tls\") pod \"4c185fa5-18ff-4caf-b853-dee1d4b0432f\" (UID: \"4c185fa5-18ff-4caf-b853-dee1d4b0432f\") "
Oct 30 20:56:05.267868 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:05.267853    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c185fa5-18ff-4caf-b853-dee1d4b0432f-kube-api-access-zgk4c\" (UniqueName: \"kubernetes.io/projected/4c185fa5-18ff-4caf-b853-dee1d4b0432f-kube-api-access-zgk4c\") pod \"4c185fa5-18ff-4caf-b853-dee1d4b0432f\" (UID: \"4c185fa5-18ff-4caf-b853-dee1d4b0432f\") "
Oct 30 20:56:05.267919 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:05.267884    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c185fa5-18ff-4caf-b853-dee1d4b0432f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c185fa5-18ff-4caf-b853-dee1d4b0432f-openshift-service-ca\") pod \"4c185fa5-18ff-4caf-b853-dee1d4b0432f\" (UID: \"4c185fa5-18ff-4caf-b853-dee1d4b0432f\") "
Oct 30 20:56:05.268682 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:05.268649    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c185fa5-18ff-4caf-b853-dee1d4b0432f-openshift-service-ca" pod "4c185fa5-18ff-4caf-b853-dee1d4b0432f" (UID: "4c185fa5-18ff-4caf-b853-dee1d4b0432f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:56:05.270156 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:05.270126    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c185fa5-18ff-4caf-b853-dee1d4b0432f-metrics-certs" pod "4c185fa5-18ff-4caf-b853-dee1d4b0432f" (UID: "4c185fa5-18ff-4caf-b853-dee1d4b0432f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:56:05.270156 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:05.270142    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c185fa5-18ff-4caf-b853-dee1d4b0432f-maas-api-tls" pod "4c185fa5-18ff-4caf-b853-dee1d4b0432f" (UID: "4c185fa5-18ff-4caf-b853-dee1d4b0432f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:56:05.270350 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:05.270305    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c185fa5-18ff-4caf-b853-dee1d4b0432f-kube-api-access-zgk4c" pod "4c185fa5-18ff-4caf-b853-dee1d4b0432f" (UID: "4c185fa5-18ff-4caf-b853-dee1d4b0432f"). InnerVolumeSpecName "kube-api-access-zgk4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:56:05.369016 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:05.368984    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgk4c\" (UniqueName: \"kubernetes.io/projected/4c185fa5-18ff-4caf-b853-dee1d4b0432f-kube-api-access-zgk4c\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:05.369016 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:05.369010    4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c185fa5-18ff-4caf-b853-dee1d4b0432f-openshift-service-ca\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:05.369016 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:05.369020    4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c185fa5-18ff-4caf-b853-dee1d4b0432f-metrics-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:05.369261 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:05.369030    4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c185fa5-18ff-4caf-b853-dee1d4b0432f-maas-api-tls\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:56:05.639485 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:05.639456    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"]
Oct 30 20:56:05.645206 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:05.645181    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-712099-6c475c4f86-8txg8"]
Oct 30 20:56:06.431558 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:06.431524    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c185fa5-18ff-4caf-b853-dee1d4b0432f" path="/var/lib/kubelet/pods/4c185fa5-18ff-4caf-b853-dee1d4b0432f/volumes"
Oct 30 20:56:06.647702 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:06.647667    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"]
Oct 30 20:56:06.648146 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:06.648132    4471 cpu_manager.go:414] "containerMap: removing container" podUID="93409cb5-1925-4012-8983-1491ead93a97" containerName="payload-pre-processing"
Oct 30 20:56:06.648201 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:06.648147    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93409cb5-1925-4012-8983-1491ead93a97" containerName="payload-pre-processing"
Oct 30 20:56:06.648201 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:06.648155    4471 cpu_manager.go:414] "containerMap: removing container" podUID="4c185fa5-18ff-4caf-b853-dee1d4b0432f" containerName="maas-api"
Oct 30 20:56:06.648201 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:06.648161    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c185fa5-18ff-4caf-b853-dee1d4b0432f" containerName="maas-api"
Oct 30 20:56:06.648295 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:06.648225    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="93409cb5-1925-4012-8983-1491ead93a97" containerName="payload-pre-processing"
Oct 30 20:56:06.648295 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:06.648233    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c185fa5-18ff-4caf-b853-dee1d4b0432f" containerName="maas-api"
Oct 30 20:56:06.652607 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:06.652590    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"
Oct 30 20:56:06.685949 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:06.685873    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0952f140-c7e9-4e47-bf2b-a43248d73b20-openshift-service-ca\") pod \"maas-api-e2e-api-b-65948c-59c988f5f-kfmmt\" (UID: \"0952f140-c7e9-4e47-bf2b-a43248d73b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"
Oct 30 20:56:06.686074 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:06.686034    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0952f140-c7e9-4e47-bf2b-a43248d73b20-metrics-certs\") pod \"maas-api-e2e-api-b-65948c-59c988f5f-kfmmt\" (UID: \"0952f140-c7e9-4e47-bf2b-a43248d73b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"
Oct 30 20:56:06.686132 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:06.686080    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0952f140-c7e9-4e47-bf2b-a43248d73b20-maas-api-tls\") pod \"maas-api-e2e-api-b-65948c-59c988f5f-kfmmt\" (UID: \"0952f140-c7e9-4e47-bf2b-a43248d73b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"
Oct 30 20:56:06.686218 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:06.686197    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjxmq\" (UniqueName: \"kubernetes.io/projected/0952f140-c7e9-4e47-bf2b-a43248d73b20-kube-api-access-bjxmq\") pod \"maas-api-e2e-api-b-65948c-59c988f5f-kfmmt\" (UID: \"0952f140-c7e9-4e47-bf2b-a43248d73b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"
Oct 30 20:56:06.756078 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:06.756048    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-65948c-serving-cert\""
Oct 30 20:56:06.758857 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:06.758833    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"]
Oct 30 20:56:06.787748 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:06.787718    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0952f140-c7e9-4e47-bf2b-a43248d73b20-openshift-service-ca\") pod \"maas-api-e2e-api-b-65948c-59c988f5f-kfmmt\" (UID: \"0952f140-c7e9-4e47-bf2b-a43248d73b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"
Oct 30 20:56:06.787892 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:06.787850    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0952f140-c7e9-4e47-bf2b-a43248d73b20-metrics-certs\") pod \"maas-api-e2e-api-b-65948c-59c988f5f-kfmmt\" (UID: \"0952f140-c7e9-4e47-bf2b-a43248d73b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"
Oct 30 20:56:06.787935 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:06.787900    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0952f140-c7e9-4e47-bf2b-a43248d73b20-maas-api-tls\") pod \"maas-api-e2e-api-b-65948c-59c988f5f-kfmmt\" (UID: \"0952f140-c7e9-4e47-bf2b-a43248d73b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"
Oct 30 20:56:06.787970 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:06.787948    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjxmq\" (UniqueName: \"kubernetes.io/projected/0952f140-c7e9-4e47-bf2b-a43248d73b20-kube-api-access-bjxmq\") pod \"maas-api-e2e-api-b-65948c-59c988f5f-kfmmt\" (UID: \"0952f140-c7e9-4e47-bf2b-a43248d73b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"
Oct 30 20:56:06.788567 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:06.788542    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0952f140-c7e9-4e47-bf2b-a43248d73b20-openshift-service-ca\") pod \"maas-api-e2e-api-b-65948c-59c988f5f-kfmmt\" (UID: \"0952f140-c7e9-4e47-bf2b-a43248d73b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"
Oct 30 20:56:06.788671 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:56:06.788648    4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-serving-cert: secret "maas-api-e2e-api-b-65948c-serving-cert" not found
Oct 30 20:56:06.788724 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:56:06.788715    4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0952f140-c7e9-4e47-bf2b-a43248d73b20-maas-api-tls podName:0952f140-c7e9-4e47-bf2b-a43248d73b20 nodeName:}" failed. No retries permitted until 2026-10-30 20:56:07.288696692 +0000 UTC m=+2186.967325299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0952f140-c7e9-4e47-bf2b-a43248d73b20-maas-api-tls") pod "maas-api-e2e-api-b-65948c-59c988f5f-kfmmt" (UID: "0952f140-c7e9-4e47-bf2b-a43248d73b20") : secret "maas-api-e2e-api-b-65948c-serving-cert" not found
Oct 30 20:56:06.791757 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:06.791506    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0952f140-c7e9-4e47-bf2b-a43248d73b20-metrics-certs\") pod \"maas-api-e2e-api-b-65948c-59c988f5f-kfmmt\" (UID: \"0952f140-c7e9-4e47-bf2b-a43248d73b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"
Oct 30 20:56:06.847668 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:06.847640    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjxmq\" (UniqueName: \"kubernetes.io/projected/0952f140-c7e9-4e47-bf2b-a43248d73b20-kube-api-access-bjxmq\") pod \"maas-api-e2e-api-b-65948c-59c988f5f-kfmmt\" (UID: \"0952f140-c7e9-4e47-bf2b-a43248d73b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"
Oct 30 20:56:07.293596 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:07.293561    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0952f140-c7e9-4e47-bf2b-a43248d73b20-maas-api-tls\") pod \"maas-api-e2e-api-b-65948c-59c988f5f-kfmmt\" (UID: \"0952f140-c7e9-4e47-bf2b-a43248d73b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"
Oct 30 20:56:07.296246 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:07.296212    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0952f140-c7e9-4e47-bf2b-a43248d73b20-maas-api-tls\") pod \"maas-api-e2e-api-b-65948c-59c988f5f-kfmmt\" (UID: \"0952f140-c7e9-4e47-bf2b-a43248d73b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"
Oct 30 20:56:07.564816 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:07.564731    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"
Oct 30 20:56:07.943100 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:07.943065    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"]
Oct 30 20:56:07.944829 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:56:07.944803    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0952f140_c7e9_4e47_bf2b_a43248d73b20.slice/crio-ab0e660d7b2486dccbbac36b1381955326bc96392fa098144905a2a4361338c1 WatchSource:0}: Error finding container ab0e660d7b2486dccbbac36b1381955326bc96392fa098144905a2a4361338c1: Status 404 returned error can't find the container with id ab0e660d7b2486dccbbac36b1381955326bc96392fa098144905a2a4361338c1
Oct 30 20:56:08.227614 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:08.227530    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt" event={"ID":"0952f140-c7e9-4e47-bf2b-a43248d73b20","Type":"ContainerStarted","Data":"ab0e660d7b2486dccbbac36b1381955326bc96392fa098144905a2a4361338c1"}
Oct 30 20:56:09.274360 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:09.272724    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt" event={"ID":"0952f140-c7e9-4e47-bf2b-a43248d73b20","Type":"ContainerStarted","Data":"86d5cb7ce36713dc1212a0a34764fa46874ecbfb8d20dfa977df302b3bc25b99"}
Oct 30 20:56:09.274360 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:09.273610    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"
Oct 30 20:56:09.391354 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:09.391273    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt" podStartSLOduration=2.573413947 podStartE2EDuration="3.391259106s" totalImagesPullingTime="817.845159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:09.388411854 +0000 UTC m=+2189.067040471" watchObservedRunningTime="2026-10-30 20:56:09.391259106 +0000 UTC m=+2189.069887721"
Oct 30 20:56:15.284084 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:15.284053    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"
Oct 30 20:56:25.550405 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:25.542390    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg"]
Oct 30 20:56:25.611868 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:25.611832    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg"
Oct 30 20:56:25.615694 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:25.615666    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg"]
Oct 30 20:56:25.670565 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:25.670538    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-cd2d96-dockercfg-q85jm\""
Oct 30 20:56:25.713402 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:25.713373    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-cd2d96\""
Oct 30 20:56:25.768783 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:25.768741    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8wcg\" (UniqueName: \"kubernetes.io/projected/da8b40b6-546e-401c-bb07-22b1fcb6791f-kube-api-access-c8wcg\") pod \"payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg\" (UID: \"da8b40b6-546e-401c-bb07-22b1fcb6791f\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg"
Oct 30 20:56:25.768966 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:25.768817    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8b40b6-546e-401c-bb07-22b1fcb6791f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg\" (UID: \"da8b40b6-546e-401c-bb07-22b1fcb6791f\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg"
Oct 30 20:56:25.870002 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:25.869913    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8wcg\" (UniqueName: \"kubernetes.io/projected/da8b40b6-546e-401c-bb07-22b1fcb6791f-kube-api-access-c8wcg\") pod \"payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg\" (UID: \"da8b40b6-546e-401c-bb07-22b1fcb6791f\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg"
Oct 30 20:56:25.870002 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:25.869963    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8b40b6-546e-401c-bb07-22b1fcb6791f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg\" (UID: \"da8b40b6-546e-401c-bb07-22b1fcb6791f\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg"
Oct 30 20:56:25.870786 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:25.870761    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8b40b6-546e-401c-bb07-22b1fcb6791f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg\" (UID: \"da8b40b6-546e-401c-bb07-22b1fcb6791f\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg"
Oct 30 20:56:25.927431 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:25.927400    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8wcg\" (UniqueName: \"kubernetes.io/projected/da8b40b6-546e-401c-bb07-22b1fcb6791f-kube-api-access-c8wcg\") pod \"payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg\" (UID: \"da8b40b6-546e-401c-bb07-22b1fcb6791f\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg"
Oct 30 20:56:25.935286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:25.935265    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg"
Oct 30 20:56:26.334605 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:26.334577    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg"]
Oct 30 20:56:26.336282 ip-10-0-138-34 kubenswrapper[4471]: W1030 20:56:26.336248    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8b40b6_546e_401c_bb07_22b1fcb6791f.slice/crio-64e517dfdcfa93df520e49c5cf56e70f29975cd722e64c21d13e8a3e4b403dcd WatchSource:0}: Error finding container 64e517dfdcfa93df520e49c5cf56e70f29975cd722e64c21d13e8a3e4b403dcd: Status 404 returned error can't find the container with id 64e517dfdcfa93df520e49c5cf56e70f29975cd722e64c21d13e8a3e4b403dcd
Oct 30 20:56:26.367997 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:26.367969    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg" event={"ID":"da8b40b6-546e-401c-bb07-22b1fcb6791f","Type":"ContainerStarted","Data":"64e517dfdcfa93df520e49c5cf56e70f29975cd722e64c21d13e8a3e4b403dcd"}
Oct 30 20:56:27.374689 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:27.374651    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg" event={"ID":"da8b40b6-546e-401c-bb07-22b1fcb6791f","Type":"ContainerStarted","Data":"e73df55ba703bff6a82b6fa16d3bbed7dc8f877ce5de301a55024b4093a302e5"}
Oct 30 20:56:27.375202 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:27.374825    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg"
Oct 30 20:56:27.543346 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:27.539469    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg" podStartSLOduration=2.539451158 podStartE2EDuration="2.539451158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:27.488687272 +0000 UTC m=+2207.167315892" watchObservedRunningTime="2026-10-30 20:56:27.539451158 +0000 UTC m=+2207.218079777"
Oct 30 20:56:30.955359 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:56:30.955303    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:56:38.383498 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:56:38.383467    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg"
Oct 30 20:57:00.954782 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:57:00.954749    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:57:05.379586 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:05.379550    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"]
Oct 30 20:57:20.401795 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.401747    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh" podUID="3dcbb5f5-93f4-4210-8274-2278fb9924ae" containerName="main" containerID="cri-o://a954e1fe27ee433244cd0569e394ac1992eba114052efc8a5097fec568fad8f4" gracePeriod=45
Oct 30 20:57:20.672286 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.672251    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:57:20.731491 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.731460    4471 generic.go:356] "Generic (PLEG): container finished" podID="3dcbb5f5-93f4-4210-8274-2278fb9924ae" containerID="a954e1fe27ee433244cd0569e394ac1992eba114052efc8a5097fec568fad8f4" exitCode=0
Oct 30 20:57:20.731689 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.731511    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh" event={"ID":"3dcbb5f5-93f4-4210-8274-2278fb9924ae","Type":"ContainerDied","Data":"a954e1fe27ee433244cd0569e394ac1992eba114052efc8a5097fec568fad8f4"}
Oct 30 20:57:20.731689 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.731545    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh" event={"ID":"3dcbb5f5-93f4-4210-8274-2278fb9924ae","Type":"ContainerDied","Data":"491a7b794191d0a14bd9e6853b3a372e7dedc0847bf1a779be75b9df69a5e7c0"}
Oct 30 20:57:20.731689 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.731562    4471 scope.go:122] "RemoveContainer" containerID="a954e1fe27ee433244cd0569e394ac1992eba114052efc8a5097fec568fad8f4"
Oct 30 20:57:20.731689 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.731514    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"
Oct 30 20:57:20.741199 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.741178    4471 scope.go:122] "RemoveContainer" containerID="a954e1fe27ee433244cd0569e394ac1992eba114052efc8a5097fec568fad8f4"
Oct 30 20:57:20.741526 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:57:20.741505    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a954e1fe27ee433244cd0569e394ac1992eba114052efc8a5097fec568fad8f4\": container with ID starting with a954e1fe27ee433244cd0569e394ac1992eba114052efc8a5097fec568fad8f4 not found: ID does not exist" containerID="a954e1fe27ee433244cd0569e394ac1992eba114052efc8a5097fec568fad8f4"
Oct 30 20:57:20.741590 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.741536    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a954e1fe27ee433244cd0569e394ac1992eba114052efc8a5097fec568fad8f4"} err="failed to get container status \"a954e1fe27ee433244cd0569e394ac1992eba114052efc8a5097fec568fad8f4\": rpc error: code = NotFound desc = could not find container \"a954e1fe27ee433244cd0569e394ac1992eba114052efc8a5097fec568fad8f4\": container with ID starting with a954e1fe27ee433244cd0569e394ac1992eba114052efc8a5097fec568fad8f4 not found: ID does not exist"
Oct 30 20:57:20.754167 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.754119    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dcbb5f5-93f4-4210-8274-2278fb9924ae-tls-certs\" (UniqueName: \"kubernetes.io/secret/3dcbb5f5-93f4-4210-8274-2278fb9924ae-tls-certs\") pod \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") "
Oct 30 20:57:20.754350 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.754214    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-model-cache\") pod \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") "
Oct 30 20:57:20.754350 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.754310    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dcbb5f5-93f4-4210-8274-2278fb9924ae-kube-api-access-thq2k\" (UniqueName: \"kubernetes.io/projected/3dcbb5f5-93f4-4210-8274-2278fb9924ae-kube-api-access-thq2k\") pod \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") "
Oct 30 20:57:20.754485 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.754383    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-tmp-dir\") pod \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") "
Oct 30 20:57:20.754485 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.754448    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-home\" (UniqueName: \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-home\") pod \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") "
Oct 30 20:57:20.754601 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.754562    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-dshm\") pod \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\" (UID: \"3dcbb5f5-93f4-4210-8274-2278fb9924ae\") "
Oct 30 20:57:20.755354 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.755285    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-home" pod "3dcbb5f5-93f4-4210-8274-2278fb9924ae" (UID: "3dcbb5f5-93f4-4210-8274-2278fb9924ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:20.755471 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.755388    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-model-cache" pod "3dcbb5f5-93f4-4210-8274-2278fb9924ae" (UID: "3dcbb5f5-93f4-4210-8274-2278fb9924ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:20.756094 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.756067    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-tmp-dir" pod "3dcbb5f5-93f4-4210-8274-2278fb9924ae" (UID: "3dcbb5f5-93f4-4210-8274-2278fb9924ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:20.757712 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.757684    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-dshm" pod "3dcbb5f5-93f4-4210-8274-2278fb9924ae" (UID: "3dcbb5f5-93f4-4210-8274-2278fb9924ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:20.757986 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.757959    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dcbb5f5-93f4-4210-8274-2278fb9924ae-kube-api-access-thq2k" pod "3dcbb5f5-93f4-4210-8274-2278fb9924ae" (UID: "3dcbb5f5-93f4-4210-8274-2278fb9924ae"). InnerVolumeSpecName "kube-api-access-thq2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:20.760914 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.760867    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dcbb5f5-93f4-4210-8274-2278fb9924ae-tls-certs" pod "3dcbb5f5-93f4-4210-8274-2278fb9924ae" (UID: "3dcbb5f5-93f4-4210-8274-2278fb9924ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:20.856142 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.856100    4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-home\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.856142 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.856145    4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-dshm\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.856367 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.856162    4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dcbb5f5-93f4-4210-8274-2278fb9924ae-tls-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.856367 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.856176    4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-model-cache\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.856367 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.856192    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thq2k\" (UniqueName: \"kubernetes.io/projected/3dcbb5f5-93f4-4210-8274-2278fb9924ae-kube-api-access-thq2k\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:20.856367 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:20.856209    4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dcbb5f5-93f4-4210-8274-2278fb9924ae-tmp-dir\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:21.160990 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:21.160956    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"]
Oct 30 20:57:21.176524 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:21.176492    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5f9b95/simulator-w1-5f9b95-kserve-644d49b84b-sj5lh"]
Oct 30 20:57:21.738344 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:21.738220    4471 generic.go:356] "Generic (PLEG): container finished" podID="675355ae-1aae-4e8e-b12f-d3727346ae51" containerID="a885dbf30b99c4d0684478872187a3679004c6545cdc93abe6ce5d4e69f9cecf" exitCode=0
Oct 30 20:57:21.738344 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:21.738286    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb" event={"ID":"675355ae-1aae-4e8e-b12f-d3727346ae51","Type":"ContainerDied","Data":"a885dbf30b99c4d0684478872187a3679004c6545cdc93abe6ce5d4e69f9cecf"}
Oct 30 20:57:21.738815 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:21.738742    4471 scope.go:122] "RemoveContainer" containerID="a885dbf30b99c4d0684478872187a3679004c6545cdc93abe6ce5d4e69f9cecf"
Oct 30 20:57:21.740379 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:21.740345    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:57:21.740476 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:21.740420    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:57:21.740476 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:21.740453    4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:57:22.434206 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:22.434177    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dcbb5f5-93f4-4210-8274-2278fb9924ae" path="/var/lib/kubelet/pods/3dcbb5f5-93f4-4210-8274-2278fb9924ae/volumes"
Oct 30 20:57:22.772976 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:22.771268    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb" event={"ID":"675355ae-1aae-4e8e-b12f-d3727346ae51","Type":"ContainerStarted","Data":"a180b359a921428e8c8775ab7f9ea8e07dd1637239c4c71b2f995435aba2a1eb"}
Oct 30 20:57:23.463347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:23.462749    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:57:23.464896 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:23.464847    4471 patch_prober.go:29] interesting pod/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 20:57:23.465186 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:23.465147    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb" podUID="675355ae-1aae-4e8e-b12f-d3727346ae51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 20:57:23.624881 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:23.624834    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"]
Oct 30 20:57:24.457834 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:24.457793    4471 patch_prober.go:29] interesting pod/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 20:57:24.458476 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:24.458441    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb" podUID="675355ae-1aae-4e8e-b12f-d3727346ae51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 20:57:24.931856 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:24.931817    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:57:25.463199 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:25.463166    4471 patch_prober.go:29] interesting pod/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 20:57:25.463828 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:25.463795    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb" podUID="675355ae-1aae-4e8e-b12f-d3727346ae51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 20:57:26.462749 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:26.462711    4471 patch_prober.go:29] interesting pod/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 20:57:26.463046 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:26.463018    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb" podUID="675355ae-1aae-4e8e-b12f-d3727346ae51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 20:57:27.462764 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:27.462723    4471 patch_prober.go:29] interesting pod/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 20:57:27.463421 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:27.463389    4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb" podUID="675355ae-1aae-4e8e-b12f-d3727346ae51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 20:57:28.462929 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:28.462902    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:57:28.464027 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:28.464009    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 20:57:30.958008 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:57:30.957964    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:57:31.381710 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:31.381673    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"]
Oct 30 20:57:38.657695 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:38.657630    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7" podUID="02b9bb13-c775-44a5-b034-563359097687" containerName="main" containerID="cri-o://f3099c762c9e2b05eee7d9271758075e246c2861d1a6c2cbce523ab0a84c90a6" gracePeriod=45
Oct 30 20:57:38.893195 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:38.893159    4471 generic.go:356] "Generic (PLEG): container finished" podID="02b9bb13-c775-44a5-b034-563359097687" containerID="f3099c762c9e2b05eee7d9271758075e246c2861d1a6c2cbce523ab0a84c90a6" exitCode=0
Oct 30 20:57:38.893417 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:38.893204    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7" event={"ID":"02b9bb13-c775-44a5-b034-563359097687","Type":"ContainerDied","Data":"f3099c762c9e2b05eee7d9271758075e246c2861d1a6c2cbce523ab0a84c90a6"}
Oct 30 20:57:38.941661 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:38.941635    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:57:38.973986 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:38.973953    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-home\" (UniqueName: \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-home\") pod \"02b9bb13-c775-44a5-b034-563359097687\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") "
Oct 30 20:57:38.974171 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:38.974002    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02b9bb13-c775-44a5-b034-563359097687-tls-certs\" (UniqueName: \"kubernetes.io/secret/02b9bb13-c775-44a5-b034-563359097687-tls-certs\") pod \"02b9bb13-c775-44a5-b034-563359097687\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") "
Oct 30 20:57:38.974171 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:38.974054    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-dshm\") pod \"02b9bb13-c775-44a5-b034-563359097687\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") "
Oct 30 20:57:38.974171 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:38.974089    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-model-cache\") pod \"02b9bb13-c775-44a5-b034-563359097687\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") "
Oct 30 20:57:38.974377 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:38.974192    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-tmp-dir\") pod \"02b9bb13-c775-44a5-b034-563359097687\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") "
Oct 30 20:57:38.974377 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:38.974207    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-home" pod "02b9bb13-c775-44a5-b034-563359097687" (UID: "02b9bb13-c775-44a5-b034-563359097687"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:38.974377 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:38.974261    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02b9bb13-c775-44a5-b034-563359097687-kube-api-access-qcls7\" (UniqueName: \"kubernetes.io/projected/02b9bb13-c775-44a5-b034-563359097687-kube-api-access-qcls7\") pod \"02b9bb13-c775-44a5-b034-563359097687\" (UID: \"02b9bb13-c775-44a5-b034-563359097687\") "
Oct 30 20:57:38.974560 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:38.974411    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-model-cache" pod "02b9bb13-c775-44a5-b034-563359097687" (UID: "02b9bb13-c775-44a5-b034-563359097687"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:38.974560 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:38.974520    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-tmp-dir" pod "02b9bb13-c775-44a5-b034-563359097687" (UID: "02b9bb13-c775-44a5-b034-563359097687"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:38.974719 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:38.974701    4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-home\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:38.974779 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:38.974721    4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-model-cache\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:38.974779 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:38.974755    4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-tmp-dir\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:38.976592 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:38.976562    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-dshm" pod "02b9bb13-c775-44a5-b034-563359097687" (UID: "02b9bb13-c775-44a5-b034-563359097687"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:38.976719 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:38.976622    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02b9bb13-c775-44a5-b034-563359097687-tls-certs" pod "02b9bb13-c775-44a5-b034-563359097687" (UID: "02b9bb13-c775-44a5-b034-563359097687"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:38.976719 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:38.976667    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b9bb13-c775-44a5-b034-563359097687-kube-api-access-qcls7" pod "02b9bb13-c775-44a5-b034-563359097687" (UID: "02b9bb13-c775-44a5-b034-563359097687"). InnerVolumeSpecName "kube-api-access-qcls7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:39.076200 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:39.076167    4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02b9bb13-c775-44a5-b034-563359097687-tls-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.076200 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:39.076203    4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b9bb13-c775-44a5-b034-563359097687-dshm\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.076421 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:39.076218    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcls7\" (UniqueName: \"kubernetes.io/projected/02b9bb13-c775-44a5-b034-563359097687-kube-api-access-qcls7\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:39.900075 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:39.900050    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"
Oct 30 20:57:39.900621 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:39.900075    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7" event={"ID":"02b9bb13-c775-44a5-b034-563359097687","Type":"ContainerDied","Data":"45482886c79faccc7986653b080aa18520de2cd8869901c4c7fcf414b82c04e6"}
Oct 30 20:57:39.900621 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:39.900129    4471 scope.go:122] "RemoveContainer" containerID="f3099c762c9e2b05eee7d9271758075e246c2861d1a6c2cbce523ab0a84c90a6"
Oct 30 20:57:40.061575 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:40.061541    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"]
Oct 30 20:57:40.110973 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:40.110938    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-a78bf4/premium-w5-a78bf4-kserve-66f5656669-5spd7"]
Oct 30 20:57:40.432437 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:40.432407    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02b9bb13-c775-44a5-b034-563359097687" path="/var/lib/kubelet/pods/02b9bb13-c775-44a5-b034-563359097687/volumes"
Oct 30 20:57:46.402600 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.402555    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7" podUID="ee5d26da-85dc-441e-86ac-87ea9512b8c5" containerName="main" containerID="cri-o://b8de4c27843359ab8efdec21ca723ff4c2f6c0930479de1a213cdb4286680a29" gracePeriod=45
Oct 30 20:57:46.481630 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.481589    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7" podUID="ee5d26da-85dc-441e-86ac-87ea9512b8c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": dial tcp 10.133.0.76:8000: connect: connection refused"
Oct 30 20:57:46.664283 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.664224    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:57:46.790623 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.790593    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee5d26da-85dc-441e-86ac-87ea9512b8c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee5d26da-85dc-441e-86ac-87ea9512b8c5-tls-certs\") pod \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") "
Oct 30 20:57:46.790822 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.790710    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5d26da-85dc-441e-86ac-87ea9512b8c5-kube-api-access-tpj64\" (UniqueName: \"kubernetes.io/projected/ee5d26da-85dc-441e-86ac-87ea9512b8c5-kube-api-access-tpj64\") pod \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") "
Oct 30 20:57:46.790902 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.790840    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-model-cache\") pod \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") "
Oct 30 20:57:46.790987 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.790968    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-dshm\") pod \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") "
Oct 30 20:57:46.791051 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.791001    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-tmp-dir\") pod \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") "
Oct 30 20:57:46.791051 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.791034    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-home\" (UniqueName: \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-home\") pod \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\" (UID: \"ee5d26da-85dc-441e-86ac-87ea9512b8c5\") "
Oct 30 20:57:46.791163 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.791103    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-model-cache" pod "ee5d26da-85dc-441e-86ac-87ea9512b8c5" (UID: "ee5d26da-85dc-441e-86ac-87ea9512b8c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:46.796353 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.791438    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-tmp-dir" pod "ee5d26da-85dc-441e-86ac-87ea9512b8c5" (UID: "ee5d26da-85dc-441e-86ac-87ea9512b8c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:46.796353 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.791643    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-home" pod "ee5d26da-85dc-441e-86ac-87ea9512b8c5" (UID: "ee5d26da-85dc-441e-86ac-87ea9512b8c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:46.796353 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.791744    4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-model-cache\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:46.796353 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.791766    4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-tmp-dir\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:46.796353 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.793565    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee5d26da-85dc-441e-86ac-87ea9512b8c5-tls-certs" pod "ee5d26da-85dc-441e-86ac-87ea9512b8c5" (UID: "ee5d26da-85dc-441e-86ac-87ea9512b8c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:46.796353 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.793607    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5d26da-85dc-441e-86ac-87ea9512b8c5-kube-api-access-tpj64" pod "ee5d26da-85dc-441e-86ac-87ea9512b8c5" (UID: "ee5d26da-85dc-441e-86ac-87ea9512b8c5"). InnerVolumeSpecName "kube-api-access-tpj64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:46.796353 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.794411    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-dshm" pod "ee5d26da-85dc-441e-86ac-87ea9512b8c5" (UID: "ee5d26da-85dc-441e-86ac-87ea9512b8c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:57:46.893233 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.893203    4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-dshm\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:46.893233 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.893232    4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee5d26da-85dc-441e-86ac-87ea9512b8c5-home\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:46.893233 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.893240    4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee5d26da-85dc-441e-86ac-87ea9512b8c5-tls-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:46.893500 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.893252    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpj64\" (UniqueName: \"kubernetes.io/projected/ee5d26da-85dc-441e-86ac-87ea9512b8c5-kube-api-access-tpj64\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:46.949599 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.949513    4471 generic.go:356] "Generic (PLEG): container finished" podID="ee5d26da-85dc-441e-86ac-87ea9512b8c5" containerID="b8de4c27843359ab8efdec21ca723ff4c2f6c0930479de1a213cdb4286680a29" exitCode=0
Oct 30 20:57:46.949599 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.949551    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"
Oct 30 20:57:46.949599 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.949584    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7" event={"ID":"ee5d26da-85dc-441e-86ac-87ea9512b8c5","Type":"ContainerDied","Data":"b8de4c27843359ab8efdec21ca723ff4c2f6c0930479de1a213cdb4286680a29"}
Oct 30 20:57:46.949832 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.949617    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7" event={"ID":"ee5d26da-85dc-441e-86ac-87ea9512b8c5","Type":"ContainerDied","Data":"d99ae266bc8d84fbb9ed53390d8ea303c4cd75074623b52e84503bc63d343889"}
Oct 30 20:57:46.949832 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.949634    4471 scope.go:122] "RemoveContainer" containerID="b8de4c27843359ab8efdec21ca723ff4c2f6c0930479de1a213cdb4286680a29"
Oct 30 20:57:46.960750 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.960722    4471 scope.go:122] "RemoveContainer" containerID="b8de4c27843359ab8efdec21ca723ff4c2f6c0930479de1a213cdb4286680a29"
Oct 30 20:57:46.961071 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:57:46.961048    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8de4c27843359ab8efdec21ca723ff4c2f6c0930479de1a213cdb4286680a29\": container with ID starting with b8de4c27843359ab8efdec21ca723ff4c2f6c0930479de1a213cdb4286680a29 not found: ID does not exist" containerID="b8de4c27843359ab8efdec21ca723ff4c2f6c0930479de1a213cdb4286680a29"
Oct 30 20:57:46.961146 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:46.961080    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8de4c27843359ab8efdec21ca723ff4c2f6c0930479de1a213cdb4286680a29"} err="failed to get container status \"b8de4c27843359ab8efdec21ca723ff4c2f6c0930479de1a213cdb4286680a29\": rpc error: code = NotFound desc = could not find container \"b8de4c27843359ab8efdec21ca723ff4c2f6c0930479de1a213cdb4286680a29\": container with ID starting with b8de4c27843359ab8efdec21ca723ff4c2f6c0930479de1a213cdb4286680a29 not found: ID does not exist"
Oct 30 20:57:47.129347 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:47.129196    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"]
Oct 30 20:57:47.141351 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:47.132848    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-27219c73/e2e-lifecycle-model-27219c73-kserve-5dffb8dcd9-fz4z7"]
Oct 30 20:57:48.431546 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:57:48.431514    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5d26da-85dc-441e-86ac-87ea9512b8c5" path="/var/lib/kubelet/pods/ee5d26da-85dc-441e-86ac-87ea9512b8c5/volumes"
Oct 30 20:58:00.963278 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:58:00.963198    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:58:30.955076 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:58:30.955041    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:58:44.407674 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:44.407639    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"]
Oct 30 20:58:44.408178 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:44.407837    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978" podUID="fa37152b-e260-4cd0-9c00-528cc696aaae" containerName="istio-proxy" containerID="cri-o://5da9e61983809800b300a6113e15c4e43297e3986c922dd1e9f497cffb3dc260" gracePeriod=30
Oct 30 20:58:46.895706 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:46.895669    4471 patch_prober.go:29] interesting pod/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:58:46.896075 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:46.895731    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978" podUID="fa37152b-e260-4cd0-9c00-528cc696aaae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:58:49.677295 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.677232    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:58:49.823141 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.823111    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa37152b-e260-4cd0-9c00-528cc696aaae-kube-api-access-mvnt2\" (UniqueName: \"kubernetes.io/projected/fa37152b-e260-4cd0-9c00-528cc696aaae-kube-api-access-mvnt2\") pod \"fa37152b-e260-4cd0-9c00-528cc696aaae\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") "
Oct 30 20:58:49.823353 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.823211    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-token\" (UniqueName: \"kubernetes.io/projected/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-token\") pod \"fa37152b-e260-4cd0-9c00-528cc696aaae\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") "
Oct 30 20:58:49.823353 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.823256    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-data\") pod \"fa37152b-e260-4cd0-9c00-528cc696aaae\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") "
Oct 30 20:58:49.823353 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.823289    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-workload-socket\") pod \"fa37152b-e260-4cd0-9c00-528cc696aaae\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") "
Oct 30 20:58:49.823353 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.823312    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-podinfo\") pod \"fa37152b-e260-4cd0-9c00-528cc696aaae\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") "
Oct 30 20:58:49.823579 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.823419    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa37152b-e260-4cd0-9c00-528cc696aaae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa37152b-e260-4cd0-9c00-528cc696aaae-istiod-ca-cert\") pod \"fa37152b-e260-4cd0-9c00-528cc696aaae\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") "
Oct 30 20:58:49.823579 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.823514    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-workload-certs\") pod \"fa37152b-e260-4cd0-9c00-528cc696aaae\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") "
Oct 30 20:58:49.823579 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.823548    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-credential-socket\") pod \"fa37152b-e260-4cd0-9c00-528cc696aaae\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") "
Oct 30 20:58:49.823731 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.823588    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-envoy\") pod \"fa37152b-e260-4cd0-9c00-528cc696aaae\" (UID: \"fa37152b-e260-4cd0-9c00-528cc696aaae\") "
Oct 30 20:58:49.823731 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.823698    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-workload-socket" pod "fa37152b-e260-4cd0-9c00-528cc696aaae" (UID: "fa37152b-e260-4cd0-9c00-528cc696aaae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:49.824246 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.823866    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-data" pod "fa37152b-e260-4cd0-9c00-528cc696aaae" (UID: "fa37152b-e260-4cd0-9c00-528cc696aaae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:49.824246 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.823965    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-workload-certs" pod "fa37152b-e260-4cd0-9c00-528cc696aaae" (UID: "fa37152b-e260-4cd0-9c00-528cc696aaae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:49.824246 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.824141    4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-data\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:49.824246 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.824179    4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-workload-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:49.824246 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.824196    4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-workload-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:49.824246 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.824200    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa37152b-e260-4cd0-9c00-528cc696aaae-istiod-ca-cert" pod "fa37152b-e260-4cd0-9c00-528cc696aaae" (UID: "fa37152b-e260-4cd0-9c00-528cc696aaae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:58:49.825161 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.825133    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-credential-socket" pod "fa37152b-e260-4cd0-9c00-528cc696aaae" (UID: "fa37152b-e260-4cd0-9c00-528cc696aaae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:49.826134 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.826111    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-token" pod "fa37152b-e260-4cd0-9c00-528cc696aaae" (UID: "fa37152b-e260-4cd0-9c00-528cc696aaae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:58:49.826235 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.826134    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa37152b-e260-4cd0-9c00-528cc696aaae-kube-api-access-mvnt2" pod "fa37152b-e260-4cd0-9c00-528cc696aaae" (UID: "fa37152b-e260-4cd0-9c00-528cc696aaae"). InnerVolumeSpecName "kube-api-access-mvnt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:58:49.826235 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.826214    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-envoy" pod "fa37152b-e260-4cd0-9c00-528cc696aaae" (UID: "fa37152b-e260-4cd0-9c00-528cc696aaae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:58:49.826456 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.826435    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-podinfo" pod "fa37152b-e260-4cd0-9c00-528cc696aaae" (UID: "fa37152b-e260-4cd0-9c00-528cc696aaae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:58:49.925094 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.925060    4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa37152b-e260-4cd0-9c00-528cc696aaae-istiod-ca-cert\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:49.925094 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.925088    4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-credential-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:49.925094 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.925098    4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-envoy\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:49.925312 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.925108    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvnt2\" (UniqueName: \"kubernetes.io/projected/fa37152b-e260-4cd0-9c00-528cc696aaae-kube-api-access-mvnt2\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:49.925312 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.925119    4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-token\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:49.925312 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:49.925128    4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa37152b-e260-4cd0-9c00-528cc696aaae-istio-podinfo\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:50.375824 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:50.375793    4471 generic.go:356] "Generic (PLEG): container finished" podID="fa37152b-e260-4cd0-9c00-528cc696aaae" containerID="5da9e61983809800b300a6113e15c4e43297e3986c922dd1e9f497cffb3dc260" exitCode=0
Oct 30 20:58:50.375992 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:50.375839    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"
Oct 30 20:58:50.375992 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:50.375868    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978" event={"ID":"fa37152b-e260-4cd0-9c00-528cc696aaae","Type":"ContainerDied","Data":"5da9e61983809800b300a6113e15c4e43297e3986c922dd1e9f497cffb3dc260"}
Oct 30 20:58:50.375992 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:50.375924    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978" event={"ID":"fa37152b-e260-4cd0-9c00-528cc696aaae","Type":"ContainerDied","Data":"eae935b02a3e29ffb9bd0831bd8a902bfde7907fa846182da372f22b1837f2f7"}
Oct 30 20:58:50.375992 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:50.375948    4471 scope.go:122] "RemoveContainer" containerID="5da9e61983809800b300a6113e15c4e43297e3986c922dd1e9f497cffb3dc260"
Oct 30 20:58:50.385806 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:50.385786    4471 scope.go:122] "RemoveContainer" containerID="5da9e61983809800b300a6113e15c4e43297e3986c922dd1e9f497cffb3dc260"
Oct 30 20:58:50.386069 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:58:50.386050    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da9e61983809800b300a6113e15c4e43297e3986c922dd1e9f497cffb3dc260\": container with ID starting with 5da9e61983809800b300a6113e15c4e43297e3986c922dd1e9f497cffb3dc260 not found: ID does not exist" containerID="5da9e61983809800b300a6113e15c4e43297e3986c922dd1e9f497cffb3dc260"
Oct 30 20:58:50.386130 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:50.386078    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da9e61983809800b300a6113e15c4e43297e3986c922dd1e9f497cffb3dc260"} err="failed to get container status \"5da9e61983809800b300a6113e15c4e43297e3986c922dd1e9f497cffb3dc260\": rpc error: code = NotFound desc = could not find container \"5da9e61983809800b300a6113e15c4e43297e3986c922dd1e9f497cffb3dc260\": container with ID starting with 5da9e61983809800b300a6113e15c4e43297e3986c922dd1e9f497cffb3dc260 not found: ID does not exist"
Oct 30 20:58:50.542259 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:50.542227    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"]
Oct 30 20:58:50.591505 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:50.591474    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f9b95-openshift-default-865b85b75-lb978"]
Oct 30 20:58:52.431727 ip-10-0-138-34 kubenswrapper[4471]: I1030 20:58:52.431693    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa37152b-e260-4cd0-9c00-528cc696aaae" path="/var/lib/kubelet/pods/fa37152b-e260-4cd0-9c00-528cc696aaae/volumes"
Oct 30 20:59:00.953531 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:59:00.953493    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 20:59:30.955017 ip-10-0-138-34 kubenswrapper[4471]: E1030 20:59:30.954924    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:00:00.189416 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.189380    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9"]
Oct 30 21:00:00.189868 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.189852    4471 cpu_manager.go:414] "containerMap: removing container" podUID="02b9bb13-c775-44a5-b034-563359097687" containerName="main"
Oct 30 21:00:00.189918 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.189870    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02b9bb13-c775-44a5-b034-563359097687" containerName="main"
Oct 30 21:00:00.189918 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.189883    4471 cpu_manager.go:414] "containerMap: removing container" podUID="fa37152b-e260-4cd0-9c00-528cc696aaae" containerName="istio-proxy"
Oct 30 21:00:00.189918 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.189888    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa37152b-e260-4cd0-9c00-528cc696aaae" containerName="istio-proxy"
Oct 30 21:00:00.189918 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.189897    4471 cpu_manager.go:414] "containerMap: removing container" podUID="3dcbb5f5-93f4-4210-8274-2278fb9924ae" containerName="main"
Oct 30 21:00:00.189918 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.189903    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dcbb5f5-93f4-4210-8274-2278fb9924ae" containerName="main"
Oct 30 21:00:00.190093 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.189926    4471 cpu_manager.go:414] "containerMap: removing container" podUID="ee5d26da-85dc-441e-86ac-87ea9512b8c5" containerName="main"
Oct 30 21:00:00.190093 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.189930    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5d26da-85dc-441e-86ac-87ea9512b8c5" containerName="main"
Oct 30 21:00:00.190093 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.189996    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5d26da-85dc-441e-86ac-87ea9512b8c5" containerName="main"
Oct 30 21:00:00.190093 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.190006    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="02b9bb13-c775-44a5-b034-563359097687" containerName="main"
Oct 30 21:00:00.190093 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.190012    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa37152b-e260-4cd0-9c00-528cc696aaae" containerName="istio-proxy"
Oct 30 21:00:00.190093 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.190019    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dcbb5f5-93f4-4210-8274-2278fb9924ae" containerName="main"
Oct 30 21:00:00.194161 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.194141    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66"]
Oct 30 21:00:00.194297 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.194273    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9"
Oct 30 21:00:00.197632 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.197616    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66"
Oct 30 21:00:00.215144 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.215120    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5"]
Oct 30 21:00:00.218957 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.218940    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5"
Oct 30 21:00:00.320262 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.320236    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkqrb\""
Oct 30 21:00:00.355555 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.355499    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c96abec-44b9-45ec-b7b4-310a2fe9a59c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9\" (UID: \"1c96abec-44b9-45ec-b7b4-310a2fe9a59c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9"
Oct 30 21:00:00.355752 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.355707    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqgwn\" (UniqueName: \"kubernetes.io/projected/ea3436e4-2f27-4ec7-a5ea-4475e0717a2b-kube-api-access-cqgwn\") pod \"maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5\" (UID: \"ea3436e4-2f27-4ec7-a5ea-4475e0717a2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5"
Oct 30 21:00:00.355752 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.355745    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da735831-f0e6-49e6-ba04-c0ed92a9cb86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66\" (UID: \"da735831-f0e6-49e6-ba04-c0ed92a9cb86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66"
Oct 30 21:00:00.355875 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.355783    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bkpg\" (UniqueName: \"kubernetes.io/projected/1c96abec-44b9-45ec-b7b4-310a2fe9a59c-kube-api-access-4bkpg\") pod \"maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9\" (UID: \"1c96abec-44b9-45ec-b7b4-310a2fe9a59c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9"
Oct 30 21:00:00.355995 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.355972    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfcvh\" (UniqueName: \"kubernetes.io/projected/da735831-f0e6-49e6-ba04-c0ed92a9cb86-kube-api-access-tfcvh\") pod \"maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66\" (UID: \"da735831-f0e6-49e6-ba04-c0ed92a9cb86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66"
Oct 30 21:00:00.356096 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.356057    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3436e4-2f27-4ec7-a5ea-4475e0717a2b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5\" (UID: \"ea3436e4-2f27-4ec7-a5ea-4475e0717a2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5"
Oct 30 21:00:00.361013 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.360985    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66"]
Oct 30 21:00:00.363451 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.363424    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5"]
Oct 30 21:00:00.419947 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.419910    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9"]
Oct 30 21:00:00.456934 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.456825    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c96abec-44b9-45ec-b7b4-310a2fe9a59c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9\" (UID: \"1c96abec-44b9-45ec-b7b4-310a2fe9a59c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9"
Oct 30 21:00:00.457097 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.456966    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqgwn\" (UniqueName: \"kubernetes.io/projected/ea3436e4-2f27-4ec7-a5ea-4475e0717a2b-kube-api-access-cqgwn\") pod \"maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5\" (UID: \"ea3436e4-2f27-4ec7-a5ea-4475e0717a2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5"
Oct 30 21:00:00.457097 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.456992    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da735831-f0e6-49e6-ba04-c0ed92a9cb86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66\" (UID: \"da735831-f0e6-49e6-ba04-c0ed92a9cb86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66"
Oct 30 21:00:00.457097 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.457019    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bkpg\" (UniqueName: \"kubernetes.io/projected/1c96abec-44b9-45ec-b7b4-310a2fe9a59c-kube-api-access-4bkpg\") pod \"maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9\" (UID: \"1c96abec-44b9-45ec-b7b4-310a2fe9a59c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9"
Oct 30 21:00:00.457270 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.457173    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfcvh\" (UniqueName: \"kubernetes.io/projected/da735831-f0e6-49e6-ba04-c0ed92a9cb86-kube-api-access-tfcvh\") pod \"maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66\" (UID: \"da735831-f0e6-49e6-ba04-c0ed92a9cb86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66"
Oct 30 21:00:00.457270 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.457257    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3436e4-2f27-4ec7-a5ea-4475e0717a2b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5\" (UID: \"ea3436e4-2f27-4ec7-a5ea-4475e0717a2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5"
Oct 30 21:00:00.457594 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.457572    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c96abec-44b9-45ec-b7b4-310a2fe9a59c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9\" (UID: \"1c96abec-44b9-45ec-b7b4-310a2fe9a59c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9"
Oct 30 21:00:00.457746 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.457724    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da735831-f0e6-49e6-ba04-c0ed92a9cb86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66\" (UID: \"da735831-f0e6-49e6-ba04-c0ed92a9cb86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66"
Oct 30 21:00:00.457824 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.457808    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3436e4-2f27-4ec7-a5ea-4475e0717a2b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5\" (UID: \"ea3436e4-2f27-4ec7-a5ea-4475e0717a2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5"
Oct 30 21:00:00.514851 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.514814    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfcvh\" (UniqueName: \"kubernetes.io/projected/da735831-f0e6-49e6-ba04-c0ed92a9cb86-kube-api-access-tfcvh\") pod \"maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66\" (UID: \"da735831-f0e6-49e6-ba04-c0ed92a9cb86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66"
Oct 30 21:00:00.514851 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.514835    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bkpg\" (UniqueName: \"kubernetes.io/projected/1c96abec-44b9-45ec-b7b4-310a2fe9a59c-kube-api-access-4bkpg\") pod \"maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9\" (UID: \"1c96abec-44b9-45ec-b7b4-310a2fe9a59c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9"
Oct 30 21:00:00.515014 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.514836    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqgwn\" (UniqueName: \"kubernetes.io/projected/ea3436e4-2f27-4ec7-a5ea-4475e0717a2b-kube-api-access-cqgwn\") pod \"maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5\" (UID: \"ea3436e4-2f27-4ec7-a5ea-4475e0717a2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5"
Oct 30 21:00:00.516508 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.516489    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66"
Oct 30 21:00:00.530936 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.530912    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5"
Oct 30 21:00:00.810153 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.810087    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9"
Oct 30 21:00:00.923870 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.923839    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66"]
Oct 30 21:00:00.925463 ip-10-0-138-34 kubenswrapper[4471]: W1030 21:00:00.925437    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda735831_f0e6_49e6_ba04_c0ed92a9cb86.slice/crio-c3f76f802603f0738fe671fc02f4c675ef403bc56a0307b1674415800e636047 WatchSource:0}: Error finding container c3f76f802603f0738fe671fc02f4c675ef403bc56a0307b1674415800e636047: Status 404 returned error can't find the container with id c3f76f802603f0738fe671fc02f4c675ef403bc56a0307b1674415800e636047
Oct 30 21:00:00.963847 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:00.963811    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5"]
Oct 30 21:00:00.966051 ip-10-0-138-34 kubenswrapper[4471]: W1030 21:00:00.966012    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3436e4_2f27_4ec7_a5ea_4475e0717a2b.slice/crio-453541cc3c4bb0cd911681d276e752c5a5370567500fe518792260df001357fb WatchSource:0}: Error finding container 453541cc3c4bb0cd911681d276e752c5a5370567500fe518792260df001357fb: Status 404 returned error can't find the container with id 453541cc3c4bb0cd911681d276e752c5a5370567500fe518792260df001357fb
Oct 30 21:00:00.967124 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:00:00.967101    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:00:01.184986 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:01.184958    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9"]
Oct 30 21:00:01.186653 ip-10-0-138-34 kubenswrapper[4471]: W1030 21:00:01.186615    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c96abec_44b9_45ec_b7b4_310a2fe9a59c.slice/crio-3fe87009dd075e71eeba9fba6381ffe5fc3d07cec18a4551fae0d413a09d84ae WatchSource:0}: Error finding container 3fe87009dd075e71eeba9fba6381ffe5fc3d07cec18a4551fae0d413a09d84ae: Status 404 returned error can't find the container with id 3fe87009dd075e71eeba9fba6381ffe5fc3d07cec18a4551fae0d413a09d84ae
Oct 30 21:00:01.913017 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:01.912985    4471 generic.go:356] "Generic (PLEG): container finished" podID="ea3436e4-2f27-4ec7-a5ea-4475e0717a2b" containerID="4e12df6146feec6afb00bcc634dfd5292855a746a58080f957cdf6b46dd48da0" exitCode=60
Oct 30 21:00:01.913446 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:01.913018    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5" event={"ID":"ea3436e4-2f27-4ec7-a5ea-4475e0717a2b","Type":"ContainerDied","Data":"4e12df6146feec6afb00bcc634dfd5292855a746a58080f957cdf6b46dd48da0"}
Oct 30 21:00:01.913446 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:01.913081    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5" event={"ID":"ea3436e4-2f27-4ec7-a5ea-4475e0717a2b","Type":"ContainerStarted","Data":"453541cc3c4bb0cd911681d276e752c5a5370567500fe518792260df001357fb"}
Oct 30 21:00:01.913446 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:01.913315    4471 scope.go:122] "RemoveContainer" containerID="4e12df6146feec6afb00bcc634dfd5292855a746a58080f957cdf6b46dd48da0"
Oct 30 21:00:01.914945 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:01.914927    4471 generic.go:356] "Generic (PLEG): container finished" podID="da735831-f0e6-49e6-ba04-c0ed92a9cb86" containerID="cef057c786e92520d5749457a466f72b120019ca10b2bb8ccf42c59080c7318d" exitCode=60
Oct 30 21:00:01.915022 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:01.915004    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66" event={"ID":"da735831-f0e6-49e6-ba04-c0ed92a9cb86","Type":"ContainerDied","Data":"cef057c786e92520d5749457a466f72b120019ca10b2bb8ccf42c59080c7318d"}
Oct 30 21:00:01.915077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:01.915052    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66" event={"ID":"da735831-f0e6-49e6-ba04-c0ed92a9cb86","Type":"ContainerStarted","Data":"c3f76f802603f0738fe671fc02f4c675ef403bc56a0307b1674415800e636047"}
Oct 30 21:00:01.915289 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:01.915272    4471 scope.go:122] "RemoveContainer" containerID="cef057c786e92520d5749457a466f72b120019ca10b2bb8ccf42c59080c7318d"
Oct 30 21:00:01.916791 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:01.916771    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9" event={"ID":"1c96abec-44b9-45ec-b7b4-310a2fe9a59c","Type":"ContainerStarted","Data":"b1f8f82f409ad7ca1bbb5b6c9d6fbff349912cbc99c223a537b0337517619a50"}
Oct 30 21:00:01.916873 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:01.916820    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9" event={"ID":"1c96abec-44b9-45ec-b7b4-310a2fe9a59c","Type":"ContainerStarted","Data":"3fe87009dd075e71eeba9fba6381ffe5fc3d07cec18a4551fae0d413a09d84ae"}
Oct 30 21:00:02.249643 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:02.249552    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9" podStartSLOduration=2.249535912 podStartE2EDuration="2.249535912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:02.248801845 +0000 UTC m=+2421.927430463" watchObservedRunningTime="2026-10-30 21:00:02.249535912 +0000 UTC m=+2421.928164528"
Oct 30 21:00:02.925440 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:02.925409    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66" event={"ID":"da735831-f0e6-49e6-ba04-c0ed92a9cb86","Type":"ContainerStarted","Data":"5d5eb64a87c33fcc5495ece52990267ec9e1668844284c25f61e0ebd2a2703d5"}
Oct 30 21:00:02.926950 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:02.926929    4471 generic.go:356] "Generic (PLEG): container finished" podID="1c96abec-44b9-45ec-b7b4-310a2fe9a59c" containerID="b1f8f82f409ad7ca1bbb5b6c9d6fbff349912cbc99c223a537b0337517619a50" exitCode=60
Oct 30 21:00:02.927029 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:02.927009    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9" event={"ID":"1c96abec-44b9-45ec-b7b4-310a2fe9a59c","Type":"ContainerDied","Data":"b1f8f82f409ad7ca1bbb5b6c9d6fbff349912cbc99c223a537b0337517619a50"}
Oct 30 21:00:02.927206 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:02.927189    4471 scope.go:122] "RemoveContainer" containerID="b1f8f82f409ad7ca1bbb5b6c9d6fbff349912cbc99c223a537b0337517619a50"
Oct 30 21:00:02.928871 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:02.928845    4471 generic.go:356] "Generic (PLEG): container finished" podID="ea3436e4-2f27-4ec7-a5ea-4475e0717a2b" containerID="a194b07619aed8f8ac8d1625ce9aefcb8bb3b499ed6ef558015c7057a27dc415" exitCode=60
Oct 30 21:00:02.928959 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:02.928924    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5" event={"ID":"ea3436e4-2f27-4ec7-a5ea-4475e0717a2b","Type":"ContainerDied","Data":"a194b07619aed8f8ac8d1625ce9aefcb8bb3b499ed6ef558015c7057a27dc415"}
Oct 30 21:00:02.929021 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:02.928986    4471 scope.go:122] "RemoveContainer" containerID="4e12df6146feec6afb00bcc634dfd5292855a746a58080f957cdf6b46dd48da0"
Oct 30 21:00:02.929183 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:02.929164    4471 scope.go:122] "RemoveContainer" containerID="a194b07619aed8f8ac8d1625ce9aefcb8bb3b499ed6ef558015c7057a27dc415"
Oct 30 21:00:02.929445 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:00:02.929425    4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5_odh-ai-gateway-infra(ea3436e4-2f27-4ec7-a5ea-4475e0717a2b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5" podUID="ea3436e4-2f27-4ec7-a5ea-4475e0717a2b"
Oct 30 21:00:03.048359 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:03.048291    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66" podStartSLOduration=3.048277937 podStartE2EDuration="3.048277937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:03.045076705 +0000 UTC m=+2422.723705334" watchObservedRunningTime="2026-10-30 21:00:03.048277937 +0000 UTC m=+2422.726906564"
Oct 30 21:00:03.937003 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:03.936970    4471 generic.go:356] "Generic (PLEG): container finished" podID="1c96abec-44b9-45ec-b7b4-310a2fe9a59c" containerID="8b6618b9ac84cd8176ab965fd6761ee3099b2399bdf949f817a255f52f53781b" exitCode=60
Oct 30 21:00:03.937498 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:03.937063    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9" event={"ID":"1c96abec-44b9-45ec-b7b4-310a2fe9a59c","Type":"ContainerDied","Data":"8b6618b9ac84cd8176ab965fd6761ee3099b2399bdf949f817a255f52f53781b"}
Oct 30 21:00:03.937498 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:03.937128    4471 scope.go:122] "RemoveContainer" containerID="b1f8f82f409ad7ca1bbb5b6c9d6fbff349912cbc99c223a537b0337517619a50"
Oct 30 21:00:03.937498 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:03.937396    4471 scope.go:122] "RemoveContainer" containerID="8b6618b9ac84cd8176ab965fd6761ee3099b2399bdf949f817a255f52f53781b"
Oct 30 21:00:03.937723 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:00:03.937697    4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9_odh-ai-gateway-infra(1c96abec-44b9-45ec-b7b4-310a2fe9a59c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9" podUID="1c96abec-44b9-45ec-b7b4-310a2fe9a59c"
Oct 30 21:00:03.939357 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:03.939338    4471 scope.go:122] "RemoveContainer" containerID="a194b07619aed8f8ac8d1625ce9aefcb8bb3b499ed6ef558015c7057a27dc415"
Oct 30 21:00:03.939585 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:00:03.939566    4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5_odh-ai-gateway-infra(ea3436e4-2f27-4ec7-a5ea-4475e0717a2b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5" podUID="ea3436e4-2f27-4ec7-a5ea-4475e0717a2b"
Oct 30 21:00:03.940978 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:03.940959    4471 generic.go:356] "Generic (PLEG): container finished" podID="da735831-f0e6-49e6-ba04-c0ed92a9cb86" containerID="5d5eb64a87c33fcc5495ece52990267ec9e1668844284c25f61e0ebd2a2703d5" exitCode=60
Oct 30 21:00:03.941081 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:03.941006    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66" event={"ID":"da735831-f0e6-49e6-ba04-c0ed92a9cb86","Type":"ContainerDied","Data":"5d5eb64a87c33fcc5495ece52990267ec9e1668844284c25f61e0ebd2a2703d5"}
Oct 30 21:00:03.941203 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:03.941191    4471 scope.go:122] "RemoveContainer" containerID="5d5eb64a87c33fcc5495ece52990267ec9e1668844284c25f61e0ebd2a2703d5"
Oct 30 21:00:03.941397 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:00:03.941380    4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66_odh-ai-gateway-infra(da735831-f0e6-49e6-ba04-c0ed92a9cb86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66" podUID="da735831-f0e6-49e6-ba04-c0ed92a9cb86"
Oct 30 21:00:03.949402 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:03.949384    4471 scope.go:122] "RemoveContainer" containerID="cef057c786e92520d5749457a466f72b120019ca10b2bb8ccf42c59080c7318d"
Oct 30 21:00:04.948753 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:04.948723    4471 scope.go:122] "RemoveContainer" containerID="8b6618b9ac84cd8176ab965fd6761ee3099b2399bdf949f817a255f52f53781b"
Oct 30 21:00:04.949236 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:00:04.949061    4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9_odh-ai-gateway-infra(1c96abec-44b9-45ec-b7b4-310a2fe9a59c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9" podUID="1c96abec-44b9-45ec-b7b4-310a2fe9a59c"
Oct 30 21:00:04.950527 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:04.950503    4471 scope.go:122] "RemoveContainer" containerID="5d5eb64a87c33fcc5495ece52990267ec9e1668844284c25f61e0ebd2a2703d5"
Oct 30 21:00:04.950764 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:00:04.950744    4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66_odh-ai-gateway-infra(da735831-f0e6-49e6-ba04-c0ed92a9cb86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66" podUID="da735831-f0e6-49e6-ba04-c0ed92a9cb86"
Oct 30 21:00:15.425430 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:15.425397    4471 scope.go:122] "RemoveContainer" containerID="5d5eb64a87c33fcc5495ece52990267ec9e1668844284c25f61e0ebd2a2703d5"
Oct 30 21:00:15.857394 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:15.857360    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"]
Oct 30 21:00:16.008261 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:16.008164    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66" event={"ID":"da735831-f0e6-49e6-ba04-c0ed92a9cb86","Type":"ContainerStarted","Data":"9dc0bf5cec7c0f59ca647a871713121f82bff96e358b18e014595b9d450ddb08"}
Oct 30 21:00:16.008460 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:16.008284    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd" podUID="c120141b-7364-4435-9533-046a104f988b" containerName="istio-proxy" containerID="cri-o://7a724c54f15da025a926991823bc597fca8fe71d721271b12bf432d83bdc5da2" gracePeriod=30
Oct 30 21:00:16.425431 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:16.425397    4471 scope.go:122] "RemoveContainer" containerID="a194b07619aed8f8ac8d1625ce9aefcb8bb3b499ed6ef558015c7057a27dc415"
Oct 30 21:00:16.425905 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:16.425498    4471 scope.go:122] "RemoveContainer" containerID="8b6618b9ac84cd8176ab965fd6761ee3099b2399bdf949f817a255f52f53781b"
Oct 30 21:00:16.952849 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:16.952763    4471 patch_prober.go:29] interesting pod/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:00:16.952849 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:16.952826    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd" podUID="c120141b-7364-4435-9533-046a104f988b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:00:17.015165 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:17.015127    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9" event={"ID":"1c96abec-44b9-45ec-b7b4-310a2fe9a59c","Type":"ContainerStarted","Data":"1eb1c7fb023609092becb2a6e994d19b16f0e5050a7285cfcb0f129c7a850684"}
Oct 30 21:00:17.016993 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:17.016971    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5" event={"ID":"ea3436e4-2f27-4ec7-a5ea-4475e0717a2b","Type":"ContainerStarted","Data":"d8a2da08365c3495ba15d2fdd490a8804badb407bb1ca78d46254f453a60c3ca"}
Oct 30 21:00:17.018829 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:17.018807    4471 generic.go:356] "Generic (PLEG): container finished" podID="da735831-f0e6-49e6-ba04-c0ed92a9cb86" containerID="9dc0bf5cec7c0f59ca647a871713121f82bff96e358b18e014595b9d450ddb08" exitCode=60
Oct 30 21:00:17.018925 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:17.018870    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66" event={"ID":"da735831-f0e6-49e6-ba04-c0ed92a9cb86","Type":"ContainerDied","Data":"9dc0bf5cec7c0f59ca647a871713121f82bff96e358b18e014595b9d450ddb08"}
Oct 30 21:00:17.018925 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:17.018907    4471 scope.go:122] "RemoveContainer" containerID="5d5eb64a87c33fcc5495ece52990267ec9e1668844284c25f61e0ebd2a2703d5"
Oct 30 21:00:17.019210 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:17.019189    4471 scope.go:122] "RemoveContainer" containerID="9dc0bf5cec7c0f59ca647a871713121f82bff96e358b18e014595b9d450ddb08"
Oct 30 21:00:17.019505 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:00:17.019472    4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66_odh-ai-gateway-infra(da735831-f0e6-49e6-ba04-c0ed92a9cb86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66" podUID="da735831-f0e6-49e6-ba04-c0ed92a9cb86"
Oct 30 21:00:17.135850 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:17.135810    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66"]
Oct 30 21:00:17.368850 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:17.368803    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5" podStartSLOduration=17.368787687 podStartE2EDuration="17.368787687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:17.366483992 +0000 UTC m=+2437.045112606" watchObservedRunningTime="2026-10-30 21:00:17.368787687 +0000 UTC m=+2437.047416302"
Oct 30 21:00:18.029595 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:18.029565    4471 generic.go:356] "Generic (PLEG): container finished" podID="1c96abec-44b9-45ec-b7b4-310a2fe9a59c" containerID="1eb1c7fb023609092becb2a6e994d19b16f0e5050a7285cfcb0f129c7a850684" exitCode=60
Oct 30 21:00:18.030109 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:18.029635    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9" event={"ID":"1c96abec-44b9-45ec-b7b4-310a2fe9a59c","Type":"ContainerDied","Data":"1eb1c7fb023609092becb2a6e994d19b16f0e5050a7285cfcb0f129c7a850684"}
Oct 30 21:00:18.030109 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:18.029692    4471 scope.go:122] "RemoveContainer" containerID="8b6618b9ac84cd8176ab965fd6761ee3099b2399bdf949f817a255f52f53781b"
Oct 30 21:00:18.030109 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:18.030085    4471 scope.go:122] "RemoveContainer" containerID="1eb1c7fb023609092becb2a6e994d19b16f0e5050a7285cfcb0f129c7a850684"
Oct 30 21:00:18.030467 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:00:18.030438    4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9_odh-ai-gateway-infra(1c96abec-44b9-45ec-b7b4-310a2fe9a59c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9" podUID="1c96abec-44b9-45ec-b7b4-310a2fe9a59c"
Oct 30 21:00:18.031839 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:18.031817    4471 generic.go:356] "Generic (PLEG): container finished" podID="ea3436e4-2f27-4ec7-a5ea-4475e0717a2b" containerID="d8a2da08365c3495ba15d2fdd490a8804badb407bb1ca78d46254f453a60c3ca" exitCode=60
Oct 30 21:00:18.031956 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:18.031891    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5" event={"ID":"ea3436e4-2f27-4ec7-a5ea-4475e0717a2b","Type":"ContainerDied","Data":"d8a2da08365c3495ba15d2fdd490a8804badb407bb1ca78d46254f453a60c3ca"}
Oct 30 21:00:18.032226 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:18.032213    4471 scope.go:122] "RemoveContainer" containerID="d8a2da08365c3495ba15d2fdd490a8804badb407bb1ca78d46254f453a60c3ca"
Oct 30 21:00:18.032485 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:00:18.032464    4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5_odh-ai-gateway-infra(ea3436e4-2f27-4ec7-a5ea-4475e0717a2b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5" podUID="ea3436e4-2f27-4ec7-a5ea-4475e0717a2b"
Oct 30 21:00:18.042555 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:18.042226    4471 scope.go:122] "RemoveContainer" containerID="a194b07619aed8f8ac8d1625ce9aefcb8bb3b499ed6ef558015c7057a27dc415"
Oct 30 21:00:18.153696 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:18.151889    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9"]
Oct 30 21:00:18.187504 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:18.187479    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66"
Oct 30 21:00:18.255315 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:18.255284    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da735831-f0e6-49e6-ba04-c0ed92a9cb86-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da735831-f0e6-49e6-ba04-c0ed92a9cb86-maas-api-service-ca\") pod \"da735831-f0e6-49e6-ba04-c0ed92a9cb86\" (UID: \"da735831-f0e6-49e6-ba04-c0ed92a9cb86\") "
Oct 30 21:00:18.255315 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:18.255345    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da735831-f0e6-49e6-ba04-c0ed92a9cb86-kube-api-access-tfcvh\" (UniqueName: \"kubernetes.io/projected/da735831-f0e6-49e6-ba04-c0ed92a9cb86-kube-api-access-tfcvh\") pod \"da735831-f0e6-49e6-ba04-c0ed92a9cb86\" (UID: \"da735831-f0e6-49e6-ba04-c0ed92a9cb86\") "
Oct 30 21:00:18.255733 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:18.255706    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da735831-f0e6-49e6-ba04-c0ed92a9cb86-maas-api-service-ca" pod "da735831-f0e6-49e6-ba04-c0ed92a9cb86" (UID: "da735831-f0e6-49e6-ba04-c0ed92a9cb86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:18.257655 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:18.257627    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da735831-f0e6-49e6-ba04-c0ed92a9cb86-kube-api-access-tfcvh" pod "da735831-f0e6-49e6-ba04-c0ed92a9cb86" (UID: "da735831-f0e6-49e6-ba04-c0ed92a9cb86"). InnerVolumeSpecName "kube-api-access-tfcvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:18.356584 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:18.356497    4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da735831-f0e6-49e6-ba04-c0ed92a9cb86-maas-api-service-ca\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:18.356584 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:18.356526    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfcvh\" (UniqueName: \"kubernetes.io/projected/da735831-f0e6-49e6-ba04-c0ed92a9cb86-kube-api-access-tfcvh\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:18.378644 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:18.378610    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5"]
Oct 30 21:00:19.027853 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.027814    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"]
Oct 30 21:00:19.028354 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.028316    4471 cpu_manager.go:414] "containerMap: removing container" podUID="da735831-f0e6-49e6-ba04-c0ed92a9cb86" containerName="cleanup"
Oct 30 21:00:19.028466 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.028355    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da735831-f0e6-49e6-ba04-c0ed92a9cb86" containerName="cleanup"
Oct 30 21:00:19.028466 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.028370    4471 cpu_manager.go:414] "containerMap: removing container" podUID="da735831-f0e6-49e6-ba04-c0ed92a9cb86" containerName="cleanup"
Oct 30 21:00:19.028466 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.028377    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da735831-f0e6-49e6-ba04-c0ed92a9cb86" containerName="cleanup"
Oct 30 21:00:19.028466 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.028393    4471 cpu_manager.go:414] "containerMap: removing container" podUID="da735831-f0e6-49e6-ba04-c0ed92a9cb86" containerName="cleanup"
Oct 30 21:00:19.028466 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.028400    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da735831-f0e6-49e6-ba04-c0ed92a9cb86" containerName="cleanup"
Oct 30 21:00:19.028708 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.028570    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="da735831-f0e6-49e6-ba04-c0ed92a9cb86" containerName="cleanup"
Oct 30 21:00:19.028708 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.028581    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="da735831-f0e6-49e6-ba04-c0ed92a9cb86" containerName="cleanup"
Oct 30 21:00:19.028708 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.028593    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="da735831-f0e6-49e6-ba04-c0ed92a9cb86" containerName="cleanup"
Oct 30 21:00:19.031949 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.031929    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:19.039127 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.039105    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66"
Oct 30 21:00:19.039241 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.039130    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66" event={"ID":"da735831-f0e6-49e6-ba04-c0ed92a9cb86","Type":"ContainerDied","Data":"c3f76f802603f0738fe671fc02f4c675ef403bc56a0307b1674415800e636047"}
Oct 30 21:00:19.039241 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.039193    4471 scope.go:122] "RemoveContainer" containerID="9dc0bf5cec7c0f59ca647a871713121f82bff96e358b18e014595b9d450ddb08"
Oct 30 21:00:19.089871 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.089511    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-11a0b0\"/\"kube-root-ca.crt\""
Oct 30 21:00:19.089871 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.089677    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-11a0b0\"/\"openshift-service-ca.crt\""
Oct 30 21:00:19.089871 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.089682    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-11a0b0\"/\"default-dockercfg-h7rlt\""
Oct 30 21:00:19.089871 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.089837    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-11a0b0\"/\"test-model-11a0b0-kserve-self-signed-certs\""
Oct 30 21:00:19.097350 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.093684    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"]
Oct 30 21:00:19.190346 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.185154    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-model-cache\") pod \"test-model-11a0b0-kserve-d94dd4cbf-b2pws\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") " pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:19.190346 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.185219    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-tls-certs\") pod \"test-model-11a0b0-kserve-d94dd4cbf-b2pws\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") " pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:19.190346 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.185269    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-tmp-dir\") pod \"test-model-11a0b0-kserve-d94dd4cbf-b2pws\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") " pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:19.190346 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.185404    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkcb2\" (UniqueName: \"kubernetes.io/projected/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-kube-api-access-tkcb2\") pod \"test-model-11a0b0-kserve-d94dd4cbf-b2pws\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") " pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:19.190346 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.185448    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-dshm\") pod \"test-model-11a0b0-kserve-d94dd4cbf-b2pws\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") " pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:19.190346 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.185521    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-home\") pod \"test-model-11a0b0-kserve-d94dd4cbf-b2pws\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") " pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:19.266957 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.266928    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9"
Oct 30 21:00:19.286445 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.286411    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-model-cache\") pod \"test-model-11a0b0-kserve-d94dd4cbf-b2pws\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") " pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:19.286560 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.286459    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-tls-certs\") pod \"test-model-11a0b0-kserve-d94dd4cbf-b2pws\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") " pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:19.286560 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.286503    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-tmp-dir\") pod \"test-model-11a0b0-kserve-d94dd4cbf-b2pws\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") " pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:19.286694 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.286597    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkcb2\" (UniqueName: \"kubernetes.io/projected/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-kube-api-access-tkcb2\") pod \"test-model-11a0b0-kserve-d94dd4cbf-b2pws\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") " pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:19.286694 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.286636    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-dshm\") pod \"test-model-11a0b0-kserve-d94dd4cbf-b2pws\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") " pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:19.286816 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.286698    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-home\") pod \"test-model-11a0b0-kserve-d94dd4cbf-b2pws\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") " pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:19.286871 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.286824    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-model-cache\") pod \"test-model-11a0b0-kserve-d94dd4cbf-b2pws\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") " pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:19.287112 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.287090    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-home\") pod \"test-model-11a0b0-kserve-d94dd4cbf-b2pws\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") " pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:19.287231 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.287210    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-tmp-dir\") pod \"test-model-11a0b0-kserve-d94dd4cbf-b2pws\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") " pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:19.289299 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.289274    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-dshm\") pod \"test-model-11a0b0-kserve-d94dd4cbf-b2pws\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") " pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:19.290333 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.290295    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-tls-certs\") pod \"test-model-11a0b0-kserve-d94dd4cbf-b2pws\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") " pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:19.316607 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.316581    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5"
Oct 30 21:00:19.343460 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.343422    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkcb2\" (UniqueName: \"kubernetes.io/projected/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-kube-api-access-tkcb2\") pod \"test-model-11a0b0-kserve-d94dd4cbf-b2pws\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") " pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:19.347083 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.347062    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:19.388760 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.387737    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c96abec-44b9-45ec-b7b4-310a2fe9a59c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c96abec-44b9-45ec-b7b4-310a2fe9a59c-maas-api-service-ca\") pod \"1c96abec-44b9-45ec-b7b4-310a2fe9a59c\" (UID: \"1c96abec-44b9-45ec-b7b4-310a2fe9a59c\") "
Oct 30 21:00:19.388760 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.387837    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea3436e4-2f27-4ec7-a5ea-4475e0717a2b-kube-api-access-cqgwn\" (UniqueName: \"kubernetes.io/projected/ea3436e4-2f27-4ec7-a5ea-4475e0717a2b-kube-api-access-cqgwn\") pod \"ea3436e4-2f27-4ec7-a5ea-4475e0717a2b\" (UID: \"ea3436e4-2f27-4ec7-a5ea-4475e0717a2b\") "
Oct 30 21:00:19.388760 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.387949    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea3436e4-2f27-4ec7-a5ea-4475e0717a2b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3436e4-2f27-4ec7-a5ea-4475e0717a2b-maas-api-service-ca\") pod \"ea3436e4-2f27-4ec7-a5ea-4475e0717a2b\" (UID: \"ea3436e4-2f27-4ec7-a5ea-4475e0717a2b\") "
Oct 30 21:00:19.388760 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.388001    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c96abec-44b9-45ec-b7b4-310a2fe9a59c-kube-api-access-4bkpg\" (UniqueName: \"kubernetes.io/projected/1c96abec-44b9-45ec-b7b4-310a2fe9a59c-kube-api-access-4bkpg\") pod \"1c96abec-44b9-45ec-b7b4-310a2fe9a59c\" (UID: \"1c96abec-44b9-45ec-b7b4-310a2fe9a59c\") "
Oct 30 21:00:19.389679 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.389449    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c96abec-44b9-45ec-b7b4-310a2fe9a59c-maas-api-service-ca" pod "1c96abec-44b9-45ec-b7b4-310a2fe9a59c" (UID: "1c96abec-44b9-45ec-b7b4-310a2fe9a59c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:19.389954 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.389920    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea3436e4-2f27-4ec7-a5ea-4475e0717a2b-maas-api-service-ca" pod "ea3436e4-2f27-4ec7-a5ea-4475e0717a2b" (UID: "ea3436e4-2f27-4ec7-a5ea-4475e0717a2b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:19.392208 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.392181    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c96abec-44b9-45ec-b7b4-310a2fe9a59c-kube-api-access-4bkpg" pod "1c96abec-44b9-45ec-b7b4-310a2fe9a59c" (UID: "1c96abec-44b9-45ec-b7b4-310a2fe9a59c"). InnerVolumeSpecName "kube-api-access-4bkpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:19.392408 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.392390    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3436e4-2f27-4ec7-a5ea-4475e0717a2b-kube-api-access-cqgwn" pod "ea3436e4-2f27-4ec7-a5ea-4475e0717a2b" (UID: "ea3436e4-2f27-4ec7-a5ea-4475e0717a2b"). InnerVolumeSpecName "kube-api-access-cqgwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:19.493247 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.493139    4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3436e4-2f27-4ec7-a5ea-4475e0717a2b-maas-api-service-ca\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:19.493247 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.493178    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bkpg\" (UniqueName: \"kubernetes.io/projected/1c96abec-44b9-45ec-b7b4-310a2fe9a59c-kube-api-access-4bkpg\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:19.493247 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.493193    4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c96abec-44b9-45ec-b7b4-310a2fe9a59c-maas-api-service-ca\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:19.493247 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.493206    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqgwn\" (UniqueName: \"kubernetes.io/projected/ea3436e4-2f27-4ec7-a5ea-4475e0717a2b-kube-api-access-cqgwn\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:19.532344 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.504649    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66"]
Oct 30 21:00:19.534000 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.533170    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-df031b-29889900-7bq66"]
Oct 30 21:00:19.728170 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:19.728145    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"]
Oct 30 21:00:19.729628 ip-10-0-138-34 kubenswrapper[4471]: W1030 21:00:19.729602    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod096c8245_e0ed_4aa9_ad32_7aca48ab6aa4.slice/crio-35dddfd3acc99012d1eda0260249f5aa16855948020890c2f8f241f96c5d681f WatchSource:0}: Error finding container 35dddfd3acc99012d1eda0260249f5aa16855948020890c2f8f241f96c5d681f: Status 404 returned error can't find the container with id 35dddfd3acc99012d1eda0260249f5aa16855948020890c2f8f241f96c5d681f
Oct 30 21:00:20.053482 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:20.053402    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9"
Oct 30 21:00:20.053937 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:20.053400    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9" event={"ID":"1c96abec-44b9-45ec-b7b4-310a2fe9a59c","Type":"ContainerDied","Data":"3fe87009dd075e71eeba9fba6381ffe5fc3d07cec18a4551fae0d413a09d84ae"}
Oct 30 21:00:20.053937 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:20.053631    4471 scope.go:122] "RemoveContainer" containerID="1eb1c7fb023609092becb2a6e994d19b16f0e5050a7285cfcb0f129c7a850684"
Oct 30 21:00:20.055070 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:20.054832    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5"
Oct 30 21:00:20.055070 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:20.054844    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5" event={"ID":"ea3436e4-2f27-4ec7-a5ea-4475e0717a2b","Type":"ContainerDied","Data":"453541cc3c4bb0cd911681d276e752c5a5370567500fe518792260df001357fb"}
Oct 30 21:00:20.056559 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:20.056533    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws" event={"ID":"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4","Type":"ContainerStarted","Data":"e64f57d0c6fd80b2aad196732196d0cbc19ef1379b7357e29492722eac82bd27"}
Oct 30 21:00:20.056689 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:20.056571    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws" event={"ID":"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4","Type":"ContainerStarted","Data":"35dddfd3acc99012d1eda0260249f5aa16855948020890c2f8f241f96c5d681f"}
Oct 30 21:00:20.064248 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:20.064224    4471 scope.go:122] "RemoveContainer" containerID="d8a2da08365c3495ba15d2fdd490a8804badb407bb1ca78d46254f453a60c3ca"
Oct 30 21:00:20.219897 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:20.219846    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws" podStartSLOduration=2.219830763 podStartE2EDuration="2.219830763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:20.217580793 +0000 UTC m=+2439.896209410" watchObservedRunningTime="2026-10-30 21:00:20.219830763 +0000 UTC m=+2439.898459380"
Oct 30 21:00:20.326880 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:20.326792    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9"]
Oct 30 21:00:20.357128 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:20.357091    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:20.374725 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:20.373930    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-4ee72d-29889900-sdhb9"]
Oct 30 21:00:20.431747 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:20.431717    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c96abec-44b9-45ec-b7b4-310a2fe9a59c" path="/var/lib/kubelet/pods/1c96abec-44b9-45ec-b7b4-310a2fe9a59c/volumes"
Oct 30 21:00:20.432132 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:20.432119    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da735831-f0e6-49e6-ba04-c0ed92a9cb86" path="/var/lib/kubelet/pods/da735831-f0e6-49e6-ba04-c0ed92a9cb86/volumes"
Oct 30 21:00:20.475552 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:20.475519    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5"]
Oct 30 21:00:20.477966 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:20.477943    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e2fc1-29889900-jl2k5"]
Oct 30 21:00:21.288560 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.288491    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 21:00:21.413944 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.413907    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-istio-envoy\") pod \"c120141b-7364-4435-9533-046a104f988b\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") "
Oct 30 21:00:21.414120 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.413954    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-istio-data\") pod \"c120141b-7364-4435-9533-046a104f988b\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") "
Oct 30 21:00:21.414120 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.414000    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c120141b-7364-4435-9533-046a104f988b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c120141b-7364-4435-9533-046a104f988b-istio-podinfo\") pod \"c120141b-7364-4435-9533-046a104f988b\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") "
Oct 30 21:00:21.414120 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.414020    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-workload-socket\") pod \"c120141b-7364-4435-9533-046a104f988b\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") "
Oct 30 21:00:21.414120 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.414041    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c120141b-7364-4435-9533-046a104f988b-kube-api-access-ktrzd\" (UniqueName: \"kubernetes.io/projected/c120141b-7364-4435-9533-046a104f988b-kube-api-access-ktrzd\") pod \"c120141b-7364-4435-9533-046a104f988b\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") "
Oct 30 21:00:21.414120 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.414108    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c120141b-7364-4435-9533-046a104f988b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c120141b-7364-4435-9533-046a104f988b-istiod-ca-cert\") pod \"c120141b-7364-4435-9533-046a104f988b\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") "
Oct 30 21:00:21.414442 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.414134    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c120141b-7364-4435-9533-046a104f988b-istio-token\" (UniqueName: \"kubernetes.io/projected/c120141b-7364-4435-9533-046a104f988b-istio-token\") pod \"c120141b-7364-4435-9533-046a104f988b\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") "
Oct 30 21:00:21.414442 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.414230    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-workload-certs\") pod \"c120141b-7364-4435-9533-046a104f988b\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") "
Oct 30 21:00:21.414442 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.414303    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-credential-socket\") pod \"c120141b-7364-4435-9533-046a104f988b\" (UID: \"c120141b-7364-4435-9533-046a104f988b\") "
Oct 30 21:00:21.414565 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.414438    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-istio-data" pod "c120141b-7364-4435-9533-046a104f988b" (UID: "c120141b-7364-4435-9533-046a104f988b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:21.414565 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.414457    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-workload-socket" pod "c120141b-7364-4435-9533-046a104f988b" (UID: "c120141b-7364-4435-9533-046a104f988b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:21.414668 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.414651    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-workload-certs" pod "c120141b-7364-4435-9533-046a104f988b" (UID: "c120141b-7364-4435-9533-046a104f988b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:21.414724 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.414663    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-credential-socket" pod "c120141b-7364-4435-9533-046a104f988b" (UID: "c120141b-7364-4435-9533-046a104f988b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:21.415020 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.414998    4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-credential-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.415090 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.415026    4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-istio-data\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.415090 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.415040    4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-workload-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.415090 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.415053    4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-workload-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.415256 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.415222    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c120141b-7364-4435-9533-046a104f988b-istiod-ca-cert" pod "c120141b-7364-4435-9533-046a104f988b" (UID: "c120141b-7364-4435-9533-046a104f988b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:21.416600 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.416574    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c120141b-7364-4435-9533-046a104f988b-istio-podinfo" pod "c120141b-7364-4435-9533-046a104f988b" (UID: "c120141b-7364-4435-9533-046a104f988b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:00:21.416717 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.416615    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-istio-envoy" pod "c120141b-7364-4435-9533-046a104f988b" (UID: "c120141b-7364-4435-9533-046a104f988b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:00:21.417062 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.417043    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c120141b-7364-4435-9533-046a104f988b-kube-api-access-ktrzd" pod "c120141b-7364-4435-9533-046a104f988b" (UID: "c120141b-7364-4435-9533-046a104f988b"). InnerVolumeSpecName "kube-api-access-ktrzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:21.417133 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.417059    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c120141b-7364-4435-9533-046a104f988b-istio-token" pod "c120141b-7364-4435-9533-046a104f988b" (UID: "c120141b-7364-4435-9533-046a104f988b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:21.445047 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.445003    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz"]
Oct 30 21:00:21.516391 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.516355    4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c120141b-7364-4435-9533-046a104f988b-istio-podinfo\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.516391 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.516385    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktrzd\" (UniqueName: \"kubernetes.io/projected/c120141b-7364-4435-9533-046a104f988b-kube-api-access-ktrzd\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.516391 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.516397    4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c120141b-7364-4435-9533-046a104f988b-istiod-ca-cert\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.516616 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.516405    4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c120141b-7364-4435-9533-046a104f988b-istio-token\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.516616 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.516414    4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c120141b-7364-4435-9533-046a104f988b-istio-envoy\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.552426 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:21.552342    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf"]
Oct 30 21:00:22.071416 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:22.071380    4471 generic.go:356] "Generic (PLEG): container finished" podID="c120141b-7364-4435-9533-046a104f988b" containerID="7a724c54f15da025a926991823bc597fca8fe71d721271b12bf432d83bdc5da2" exitCode=0
Oct 30 21:00:22.071602 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:22.071432    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"
Oct 30 21:00:22.071602 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:22.071439    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd" event={"ID":"c120141b-7364-4435-9533-046a104f988b","Type":"ContainerDied","Data":"7a724c54f15da025a926991823bc597fca8fe71d721271b12bf432d83bdc5da2"}
Oct 30 21:00:22.071602 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:22.071514    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd" event={"ID":"c120141b-7364-4435-9533-046a104f988b","Type":"ContainerDied","Data":"f28f988eb6a8c3c7d967c089cf689610e687f8058694354ac59982c400150164"}
Oct 30 21:00:22.071602 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:22.071546    4471 scope.go:122] "RemoveContainer" containerID="7a724c54f15da025a926991823bc597fca8fe71d721271b12bf432d83bdc5da2"
Oct 30 21:00:22.081783 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:22.081762    4471 scope.go:122] "RemoveContainer" containerID="7a724c54f15da025a926991823bc597fca8fe71d721271b12bf432d83bdc5da2"
Oct 30 21:00:22.082066 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:00:22.082047    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a724c54f15da025a926991823bc597fca8fe71d721271b12bf432d83bdc5da2\": container with ID starting with 7a724c54f15da025a926991823bc597fca8fe71d721271b12bf432d83bdc5da2 not found: ID does not exist" containerID="7a724c54f15da025a926991823bc597fca8fe71d721271b12bf432d83bdc5da2"
Oct 30 21:00:22.082115 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:22.082075    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a724c54f15da025a926991823bc597fca8fe71d721271b12bf432d83bdc5da2"} err="failed to get container status \"7a724c54f15da025a926991823bc597fca8fe71d721271b12bf432d83bdc5da2\": rpc error: code = NotFound desc = could not find container \"7a724c54f15da025a926991823bc597fca8fe71d721271b12bf432d83bdc5da2\": container with ID starting with 7a724c54f15da025a926991823bc597fca8fe71d721271b12bf432d83bdc5da2 not found: ID does not exist"
Oct 30 21:00:22.236568 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:22.236532    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"]
Oct 30 21:00:22.241878 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:22.241853    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0a78ac-openshift-default-854db56669-frxjd"]
Oct 30 21:00:22.431668 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:22.431638    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c120141b-7364-4435-9533-046a104f988b" path="/var/lib/kubelet/pods/c120141b-7364-4435-9533-046a104f988b/volumes"
Oct 30 21:00:22.432041 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:22.432030    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea3436e4-2f27-4ec7-a5ea-4475e0717a2b" path="/var/lib/kubelet/pods/ea3436e4-2f27-4ec7-a5ea-4475e0717a2b/volumes"
Oct 30 21:00:26.445819 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:26.445779    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz" podUID="aff6334a-529c-411f-ae61-db2d2834906d" containerName="payload-processing" containerID="cri-o://f4cd4ae522050e810882c0b2250e46f70e49e2344ae4c3b5634a43dacf91dd67" gracePeriod=25
Oct 30 21:00:26.553468 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:26.553404    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf" podUID="39317fa6-714f-4826-9a67-f16b11f3f7e7" containerName="payload-pre-processing" containerID="cri-o://8f221c4a41765a8dfb4ec6552a8f1a9b63094f2d8c83fda2dd4de69b08c3535c" gracePeriod=25
Oct 30 21:00:26.731763 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:26.731731    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz"
Oct 30 21:00:26.849887 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:26.849855    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf"
Oct 30 21:00:26.876251 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:26.876224    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff6334a-529c-411f-ae61-db2d2834906d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff6334a-529c-411f-ae61-db2d2834906d-plugins-config-volume\") pod \"aff6334a-529c-411f-ae61-db2d2834906d\" (UID: \"aff6334a-529c-411f-ae61-db2d2834906d\") "
Oct 30 21:00:26.876414 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:26.876268    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff6334a-529c-411f-ae61-db2d2834906d-kube-api-access-8qlgf\" (UniqueName: \"kubernetes.io/projected/aff6334a-529c-411f-ae61-db2d2834906d-kube-api-access-8qlgf\") pod \"aff6334a-529c-411f-ae61-db2d2834906d\" (UID: \"aff6334a-529c-411f-ae61-db2d2834906d\") "
Oct 30 21:00:26.876718 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:26.876684    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff6334a-529c-411f-ae61-db2d2834906d-plugins-config-volume" pod "aff6334a-529c-411f-ae61-db2d2834906d" (UID: "aff6334a-529c-411f-ae61-db2d2834906d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:26.876905 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:26.876884    4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff6334a-529c-411f-ae61-db2d2834906d-plugins-config-volume\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:26.878729 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:26.878703    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff6334a-529c-411f-ae61-db2d2834906d-kube-api-access-8qlgf" pod "aff6334a-529c-411f-ae61-db2d2834906d" (UID: "aff6334a-529c-411f-ae61-db2d2834906d"). InnerVolumeSpecName "kube-api-access-8qlgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:26.977605 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:26.977522    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39317fa6-714f-4826-9a67-f16b11f3f7e7-kube-api-access-s6kkf\" (UniqueName: \"kubernetes.io/projected/39317fa6-714f-4826-9a67-f16b11f3f7e7-kube-api-access-s6kkf\") pod \"39317fa6-714f-4826-9a67-f16b11f3f7e7\" (UID: \"39317fa6-714f-4826-9a67-f16b11f3f7e7\") "
Oct 30 21:00:26.977766 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:26.977618    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39317fa6-714f-4826-9a67-f16b11f3f7e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39317fa6-714f-4826-9a67-f16b11f3f7e7-plugins-config-volume\") pod \"39317fa6-714f-4826-9a67-f16b11f3f7e7\" (UID: \"39317fa6-714f-4826-9a67-f16b11f3f7e7\") "
Oct 30 21:00:26.978053 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:26.978033    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qlgf\" (UniqueName: \"kubernetes.io/projected/aff6334a-529c-411f-ae61-db2d2834906d-kube-api-access-8qlgf\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:26.978152 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:26.978028    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39317fa6-714f-4826-9a67-f16b11f3f7e7-plugins-config-volume" pod "39317fa6-714f-4826-9a67-f16b11f3f7e7" (UID: "39317fa6-714f-4826-9a67-f16b11f3f7e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:26.979913 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:26.979887    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39317fa6-714f-4826-9a67-f16b11f3f7e7-kube-api-access-s6kkf" pod "39317fa6-714f-4826-9a67-f16b11f3f7e7" (UID: "39317fa6-714f-4826-9a67-f16b11f3f7e7"). InnerVolumeSpecName "kube-api-access-s6kkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:27.079589 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:27.079554    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6kkf\" (UniqueName: \"kubernetes.io/projected/39317fa6-714f-4826-9a67-f16b11f3f7e7-kube-api-access-s6kkf\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:27.079589 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:27.079582    4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39317fa6-714f-4826-9a67-f16b11f3f7e7-plugins-config-volume\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:27.101006 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:27.100975    4471 generic.go:356] "Generic (PLEG): container finished" podID="39317fa6-714f-4826-9a67-f16b11f3f7e7" containerID="8f221c4a41765a8dfb4ec6552a8f1a9b63094f2d8c83fda2dd4de69b08c3535c" exitCode=0
Oct 30 21:00:27.101120 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:27.101021    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf"
Oct 30 21:00:27.101120 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:27.101064    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf" event={"ID":"39317fa6-714f-4826-9a67-f16b11f3f7e7","Type":"ContainerDied","Data":"8f221c4a41765a8dfb4ec6552a8f1a9b63094f2d8c83fda2dd4de69b08c3535c"}
Oct 30 21:00:27.101120 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:27.101115    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf" event={"ID":"39317fa6-714f-4826-9a67-f16b11f3f7e7","Type":"ContainerDied","Data":"6b2481c000e4768a54cdd601a854387c18a523c1a61570895710646a10bf092d"}
Oct 30 21:00:27.101248 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:27.101141    4471 scope.go:122] "RemoveContainer" containerID="8f221c4a41765a8dfb4ec6552a8f1a9b63094f2d8c83fda2dd4de69b08c3535c"
Oct 30 21:00:27.102380 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:27.102359    4471 generic.go:356] "Generic (PLEG): container finished" podID="aff6334a-529c-411f-ae61-db2d2834906d" containerID="f4cd4ae522050e810882c0b2250e46f70e49e2344ae4c3b5634a43dacf91dd67" exitCode=0
Oct 30 21:00:27.102471 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:27.102437    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz"
Oct 30 21:00:27.102471 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:27.102457    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz" event={"ID":"aff6334a-529c-411f-ae61-db2d2834906d","Type":"ContainerDied","Data":"f4cd4ae522050e810882c0b2250e46f70e49e2344ae4c3b5634a43dacf91dd67"}
Oct 30 21:00:27.102549 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:27.102497    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz" event={"ID":"aff6334a-529c-411f-ae61-db2d2834906d","Type":"ContainerDied","Data":"851b78b329b4b64dc7524efe396bb4cbd0483a530c26f03fe3661d46138fc98c"}
Oct 30 21:00:27.110447 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:27.110413    4471 scope.go:122] "RemoveContainer" containerID="8f221c4a41765a8dfb4ec6552a8f1a9b63094f2d8c83fda2dd4de69b08c3535c"
Oct 30 21:00:27.110722 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:00:27.110704    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f221c4a41765a8dfb4ec6552a8f1a9b63094f2d8c83fda2dd4de69b08c3535c\": container with ID starting with 8f221c4a41765a8dfb4ec6552a8f1a9b63094f2d8c83fda2dd4de69b08c3535c not found: ID does not exist" containerID="8f221c4a41765a8dfb4ec6552a8f1a9b63094f2d8c83fda2dd4de69b08c3535c"
Oct 30 21:00:27.110805 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:27.110733    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f221c4a41765a8dfb4ec6552a8f1a9b63094f2d8c83fda2dd4de69b08c3535c"} err="failed to get container status \"8f221c4a41765a8dfb4ec6552a8f1a9b63094f2d8c83fda2dd4de69b08c3535c\": rpc error: code = NotFound desc = could not find container \"8f221c4a41765a8dfb4ec6552a8f1a9b63094f2d8c83fda2dd4de69b08c3535c\": container with ID starting with 8f221c4a41765a8dfb4ec6552a8f1a9b63094f2d8c83fda2dd4de69b08c3535c not found: ID does not exist"
Oct 30 21:00:27.110805 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:27.110756    4471 scope.go:122] "RemoveContainer" containerID="f4cd4ae522050e810882c0b2250e46f70e49e2344ae4c3b5634a43dacf91dd67"
Oct 30 21:00:27.120604 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:27.120588    4471 scope.go:122] "RemoveContainer" containerID="f4cd4ae522050e810882c0b2250e46f70e49e2344ae4c3b5634a43dacf91dd67"
Oct 30 21:00:27.120897 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:00:27.120877    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4cd4ae522050e810882c0b2250e46f70e49e2344ae4c3b5634a43dacf91dd67\": container with ID starting with f4cd4ae522050e810882c0b2250e46f70e49e2344ae4c3b5634a43dacf91dd67 not found: ID does not exist" containerID="f4cd4ae522050e810882c0b2250e46f70e49e2344ae4c3b5634a43dacf91dd67"
Oct 30 21:00:27.120974 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:27.120903    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4cd4ae522050e810882c0b2250e46f70e49e2344ae4c3b5634a43dacf91dd67"} err="failed to get container status \"f4cd4ae522050e810882c0b2250e46f70e49e2344ae4c3b5634a43dacf91dd67\": rpc error: code = NotFound desc = could not find container \"f4cd4ae522050e810882c0b2250e46f70e49e2344ae4c3b5634a43dacf91dd67\": container with ID starting with f4cd4ae522050e810882c0b2250e46f70e49e2344ae4c3b5634a43dacf91dd67 not found: ID does not exist"
Oct 30 21:00:27.274962 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:27.274884    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz"]
Oct 30 21:00:27.281469 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:27.281444    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5dc13e-84fc7d8688-ctttz"]
Oct 30 21:00:27.390459 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:27.390425    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf"]
Oct 30 21:00:27.393902 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:27.393878    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5dc13e-75d465b6f5-fh8qf"]
Oct 30 21:00:28.431568 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:28.431532    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39317fa6-714f-4826-9a67-f16b11f3f7e7" path="/var/lib/kubelet/pods/39317fa6-714f-4826-9a67-f16b11f3f7e7/volumes"
Oct 30 21:00:28.431939 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:28.431897    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff6334a-529c-411f-ae61-db2d2834906d" path="/var/lib/kubelet/pods/aff6334a-529c-411f-ae61-db2d2834906d/volumes"
Oct 30 21:00:29.347596 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:29.347558    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:29.360499 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:29.360474    4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:30.180365 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:30.179198    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:00:30.955704 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:00:30.955664    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:00:54.030893 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:54.030811    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg"]
Oct 30 21:00:59.031908 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:59.031856    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg" podUID="da8b40b6-546e-401c-bb07-22b1fcb6791f" containerName="payload-processing" containerID="cri-o://e73df55ba703bff6a82b6fa16d3bbed7dc8f877ce5de301a55024b4093a302e5" gracePeriod=25
Oct 30 21:00:59.296716 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:59.296653    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg"
Oct 30 21:00:59.349808 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:59.349768    4471 generic.go:356] "Generic (PLEG): container finished" podID="da8b40b6-546e-401c-bb07-22b1fcb6791f" containerID="e73df55ba703bff6a82b6fa16d3bbed7dc8f877ce5de301a55024b4093a302e5" exitCode=0
Oct 30 21:00:59.350019 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:59.349958    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg"
Oct 30 21:00:59.350019 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:59.349987    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg" event={"ID":"da8b40b6-546e-401c-bb07-22b1fcb6791f","Type":"ContainerDied","Data":"e73df55ba703bff6a82b6fa16d3bbed7dc8f877ce5de301a55024b4093a302e5"}
Oct 30 21:00:59.350127 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:59.350078    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg" event={"ID":"da8b40b6-546e-401c-bb07-22b1fcb6791f","Type":"ContainerDied","Data":"64e517dfdcfa93df520e49c5cf56e70f29975cd722e64c21d13e8a3e4b403dcd"}
Oct 30 21:00:59.350127 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:59.350116    4471 scope.go:122] "RemoveContainer" containerID="e73df55ba703bff6a82b6fa16d3bbed7dc8f877ce5de301a55024b4093a302e5"
Oct 30 21:00:59.364623 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:59.364604    4471 scope.go:122] "RemoveContainer" containerID="e73df55ba703bff6a82b6fa16d3bbed7dc8f877ce5de301a55024b4093a302e5"
Oct 30 21:00:59.364947 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:00:59.364923    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e73df55ba703bff6a82b6fa16d3bbed7dc8f877ce5de301a55024b4093a302e5\": container with ID starting with e73df55ba703bff6a82b6fa16d3bbed7dc8f877ce5de301a55024b4093a302e5 not found: ID does not exist" containerID="e73df55ba703bff6a82b6fa16d3bbed7dc8f877ce5de301a55024b4093a302e5"
Oct 30 21:00:59.364989 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:59.364956    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e73df55ba703bff6a82b6fa16d3bbed7dc8f877ce5de301a55024b4093a302e5"} err="failed to get container status \"e73df55ba703bff6a82b6fa16d3bbed7dc8f877ce5de301a55024b4093a302e5\": rpc error: code = NotFound desc = could not find container \"e73df55ba703bff6a82b6fa16d3bbed7dc8f877ce5de301a55024b4093a302e5\": container with ID starting with e73df55ba703bff6a82b6fa16d3bbed7dc8f877ce5de301a55024b4093a302e5 not found: ID does not exist"
Oct 30 21:00:59.408537 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:59.408503    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da8b40b6-546e-401c-bb07-22b1fcb6791f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8b40b6-546e-401c-bb07-22b1fcb6791f-plugins-config-volume\") pod \"da8b40b6-546e-401c-bb07-22b1fcb6791f\" (UID: \"da8b40b6-546e-401c-bb07-22b1fcb6791f\") "
Oct 30 21:00:59.408694 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:59.408550    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8b40b6-546e-401c-bb07-22b1fcb6791f-kube-api-access-c8wcg\" (UniqueName: \"kubernetes.io/projected/da8b40b6-546e-401c-bb07-22b1fcb6791f-kube-api-access-c8wcg\") pod \"da8b40b6-546e-401c-bb07-22b1fcb6791f\" (UID: \"da8b40b6-546e-401c-bb07-22b1fcb6791f\") "
Oct 30 21:00:59.408911 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:59.408879    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da8b40b6-546e-401c-bb07-22b1fcb6791f-plugins-config-volume" pod "da8b40b6-546e-401c-bb07-22b1fcb6791f" (UID: "da8b40b6-546e-401c-bb07-22b1fcb6791f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:59.409066 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:59.409051    4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8b40b6-546e-401c-bb07-22b1fcb6791f-plugins-config-volume\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:59.410785 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:59.410759    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8b40b6-546e-401c-bb07-22b1fcb6791f-kube-api-access-c8wcg" pod "da8b40b6-546e-401c-bb07-22b1fcb6791f" (UID: "da8b40b6-546e-401c-bb07-22b1fcb6791f"). InnerVolumeSpecName "kube-api-access-c8wcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:59.510474 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:59.510237    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8wcg\" (UniqueName: \"kubernetes.io/projected/da8b40b6-546e-401c-bb07-22b1fcb6791f-kube-api-access-c8wcg\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:59.768923 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:59.768881    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg"]
Oct 30 21:00:59.771359 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:00:59.771314    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd2d96-867bfcdccd-fwsjg"]
Oct 30 21:01:00.432583 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:00.432547    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da8b40b6-546e-401c-bb07-22b1fcb6791f" path="/var/lib/kubelet/pods/da8b40b6-546e-401c-bb07-22b1fcb6791f/volumes"
Oct 30 21:01:00.954579 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:01:00.954538    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:01:10.269683 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:10.269643    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"]
Oct 30 21:01:10.270216 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:10.269889    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt" podUID="0952f140-c7e9-4e47-bf2b-a43248d73b20" containerName="maas-api" containerID="cri-o://86d5cb7ce36713dc1212a0a34764fa46874ecbfb8d20dfa977df302b3bc25b99" gracePeriod=30
Oct 30 21:01:10.278705 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:10.278666    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt" podUID="0952f140-c7e9-4e47-bf2b-a43248d73b20" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.93:8443/health\": dial tcp 10.133.0.93:8443: connect: connection refused"
Oct 30 21:01:10.410615 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:10.410580    4471 generic.go:356] "Generic (PLEG): container finished" podID="0952f140-c7e9-4e47-bf2b-a43248d73b20" containerID="86d5cb7ce36713dc1212a0a34764fa46874ecbfb8d20dfa977df302b3bc25b99" exitCode=0
Oct 30 21:01:10.410760 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:10.410653    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt" event={"ID":"0952f140-c7e9-4e47-bf2b-a43248d73b20","Type":"ContainerDied","Data":"86d5cb7ce36713dc1212a0a34764fa46874ecbfb8d20dfa977df302b3bc25b99"}
Oct 30 21:01:10.598020 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:10.597994    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"
Oct 30 21:01:10.742348 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:10.741903    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0952f140-c7e9-4e47-bf2b-a43248d73b20-kube-api-access-bjxmq\" (UniqueName: \"kubernetes.io/projected/0952f140-c7e9-4e47-bf2b-a43248d73b20-kube-api-access-bjxmq\") pod \"0952f140-c7e9-4e47-bf2b-a43248d73b20\" (UID: \"0952f140-c7e9-4e47-bf2b-a43248d73b20\") "
Oct 30 21:01:10.742348 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:10.742025    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0952f140-c7e9-4e47-bf2b-a43248d73b20-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0952f140-c7e9-4e47-bf2b-a43248d73b20-maas-api-tls\") pod \"0952f140-c7e9-4e47-bf2b-a43248d73b20\" (UID: \"0952f140-c7e9-4e47-bf2b-a43248d73b20\") "
Oct 30 21:01:10.742348 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:10.742237    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0952f140-c7e9-4e47-bf2b-a43248d73b20-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0952f140-c7e9-4e47-bf2b-a43248d73b20-openshift-service-ca\") pod \"0952f140-c7e9-4e47-bf2b-a43248d73b20\" (UID: \"0952f140-c7e9-4e47-bf2b-a43248d73b20\") "
Oct 30 21:01:10.742348 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:10.742307    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0952f140-c7e9-4e47-bf2b-a43248d73b20-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0952f140-c7e9-4e47-bf2b-a43248d73b20-metrics-certs\") pod \"0952f140-c7e9-4e47-bf2b-a43248d73b20\" (UID: \"0952f140-c7e9-4e47-bf2b-a43248d73b20\") "
Oct 30 21:01:10.742700 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:10.742657    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0952f140-c7e9-4e47-bf2b-a43248d73b20-openshift-service-ca" pod "0952f140-c7e9-4e47-bf2b-a43248d73b20" (UID: "0952f140-c7e9-4e47-bf2b-a43248d73b20"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:10.743347 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:10.742781    4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0952f140-c7e9-4e47-bf2b-a43248d73b20-openshift-service-ca\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:10.770174 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:10.769895    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0952f140-c7e9-4e47-bf2b-a43248d73b20-maas-api-tls" pod "0952f140-c7e9-4e47-bf2b-a43248d73b20" (UID: "0952f140-c7e9-4e47-bf2b-a43248d73b20"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:10.770174 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:10.770114    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0952f140-c7e9-4e47-bf2b-a43248d73b20-metrics-certs" pod "0952f140-c7e9-4e47-bf2b-a43248d73b20" (UID: "0952f140-c7e9-4e47-bf2b-a43248d73b20"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:10.774126 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:10.774088    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0952f140-c7e9-4e47-bf2b-a43248d73b20-kube-api-access-bjxmq" pod "0952f140-c7e9-4e47-bf2b-a43248d73b20" (UID: "0952f140-c7e9-4e47-bf2b-a43248d73b20"). InnerVolumeSpecName "kube-api-access-bjxmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:10.850470 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:10.847986    4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0952f140-c7e9-4e47-bf2b-a43248d73b20-maas-api-tls\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:10.850470 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:10.848030    4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0952f140-c7e9-4e47-bf2b-a43248d73b20-metrics-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:10.850470 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:10.848048    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjxmq\" (UniqueName: \"kubernetes.io/projected/0952f140-c7e9-4e47-bf2b-a43248d73b20-kube-api-access-bjxmq\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:11.416686 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:11.416653    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"
Oct 30 21:01:11.417132 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:11.416683    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt" event={"ID":"0952f140-c7e9-4e47-bf2b-a43248d73b20","Type":"ContainerDied","Data":"ab0e660d7b2486dccbbac36b1381955326bc96392fa098144905a2a4361338c1"}
Oct 30 21:01:11.417132 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:11.416747    4471 scope.go:122] "RemoveContainer" containerID="86d5cb7ce36713dc1212a0a34764fa46874ecbfb8d20dfa977df302b3bc25b99"
Oct 30 21:01:11.579398 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:11.579276    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"]
Oct 30 21:01:11.628305 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:11.628271    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65948c-59c988f5f-kfmmt"]
Oct 30 21:01:12.431068 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:12.431035    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0952f140-c7e9-4e47-bf2b-a43248d73b20" path="/var/lib/kubelet/pods/0952f140-c7e9-4e47-bf2b-a43248d73b20/volumes"
Oct 30 21:01:30.232649 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:30.232610    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"]
Oct 30 21:01:30.233149 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:30.232848    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb" podUID="675355ae-1aae-4e8e-b12f-d3727346ae51" containerName="istio-proxy" containerID="cri-o://a180b359a921428e8c8775ab7f9ea8e07dd1637239c4c71b2f995435aba2a1eb" gracePeriod=30
Oct 30 21:01:30.954135 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:01:30.954100    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:01:35.489192 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.489169    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 21:01:35.566528 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.566438    4471 generic.go:356] "Generic (PLEG): container finished" podID="675355ae-1aae-4e8e-b12f-d3727346ae51" containerID="a180b359a921428e8c8775ab7f9ea8e07dd1637239c4c71b2f995435aba2a1eb" exitCode=0
Oct 30 21:01:35.566528 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.566495    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"
Oct 30 21:01:35.566708 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.566525    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb" event={"ID":"675355ae-1aae-4e8e-b12f-d3727346ae51","Type":"ContainerDied","Data":"a180b359a921428e8c8775ab7f9ea8e07dd1637239c4c71b2f995435aba2a1eb"}
Oct 30 21:01:35.566708 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.566580    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb" event={"ID":"675355ae-1aae-4e8e-b12f-d3727346ae51","Type":"ContainerDied","Data":"ff0e1307b3496b07156caa182902dcd27ec75b8b4f6299f3852fb18dce23f00c"}
Oct 30 21:01:35.566708 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.566606    4471 scope.go:122] "RemoveContainer" containerID="a180b359a921428e8c8775ab7f9ea8e07dd1637239c4c71b2f995435aba2a1eb"
Oct 30 21:01:35.576493 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.576010    4471 scope.go:122] "RemoveContainer" containerID="a885dbf30b99c4d0684478872187a3679004c6545cdc93abe6ce5d4e69f9cecf"
Oct 30 21:01:35.587300 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.587268    4471 scope.go:122] "RemoveContainer" containerID="a180b359a921428e8c8775ab7f9ea8e07dd1637239c4c71b2f995435aba2a1eb"
Oct 30 21:01:35.587753 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:01:35.587722    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a180b359a921428e8c8775ab7f9ea8e07dd1637239c4c71b2f995435aba2a1eb\": container with ID starting with a180b359a921428e8c8775ab7f9ea8e07dd1637239c4c71b2f995435aba2a1eb not found: ID does not exist" containerID="a180b359a921428e8c8775ab7f9ea8e07dd1637239c4c71b2f995435aba2a1eb"
Oct 30 21:01:35.587825 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.587765    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a180b359a921428e8c8775ab7f9ea8e07dd1637239c4c71b2f995435aba2a1eb"} err="failed to get container status \"a180b359a921428e8c8775ab7f9ea8e07dd1637239c4c71b2f995435aba2a1eb\": rpc error: code = NotFound desc = could not find container \"a180b359a921428e8c8775ab7f9ea8e07dd1637239c4c71b2f995435aba2a1eb\": container with ID starting with a180b359a921428e8c8775ab7f9ea8e07dd1637239c4c71b2f995435aba2a1eb not found: ID does not exist"
Oct 30 21:01:35.587825 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.587794    4471 scope.go:122] "RemoveContainer" containerID="a885dbf30b99c4d0684478872187a3679004c6545cdc93abe6ce5d4e69f9cecf"
Oct 30 21:01:35.588099 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:01:35.588080    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a885dbf30b99c4d0684478872187a3679004c6545cdc93abe6ce5d4e69f9cecf\": container with ID starting with a885dbf30b99c4d0684478872187a3679004c6545cdc93abe6ce5d4e69f9cecf not found: ID does not exist" containerID="a885dbf30b99c4d0684478872187a3679004c6545cdc93abe6ce5d4e69f9cecf"
Oct 30 21:01:35.588145 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.588106    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a885dbf30b99c4d0684478872187a3679004c6545cdc93abe6ce5d4e69f9cecf"} err="failed to get container status \"a885dbf30b99c4d0684478872187a3679004c6545cdc93abe6ce5d4e69f9cecf\": rpc error: code = NotFound desc = could not find container \"a885dbf30b99c4d0684478872187a3679004c6545cdc93abe6ce5d4e69f9cecf\": container with ID starting with a885dbf30b99c4d0684478872187a3679004c6545cdc93abe6ce5d4e69f9cecf not found: ID does not exist"
Oct 30 21:01:35.676508 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.676473    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-data\") pod \"675355ae-1aae-4e8e-b12f-d3727346ae51\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") "
Oct 30 21:01:35.676690 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.676546    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/675355ae-1aae-4e8e-b12f-d3727346ae51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/675355ae-1aae-4e8e-b12f-d3727346ae51-istiod-ca-cert\") pod \"675355ae-1aae-4e8e-b12f-d3727346ae51\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") "
Oct 30 21:01:35.676690 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.676582    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-workload-socket\") pod \"675355ae-1aae-4e8e-b12f-d3727346ae51\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") "
Oct 30 21:01:35.676690 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.676615    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/675355ae-1aae-4e8e-b12f-d3727346ae51-kube-api-access-rmw2j\" (UniqueName: \"kubernetes.io/projected/675355ae-1aae-4e8e-b12f-d3727346ae51-kube-api-access-rmw2j\") pod \"675355ae-1aae-4e8e-b12f-d3727346ae51\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") "
Oct 30 21:01:35.676690 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.676675    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-credential-socket\") pod \"675355ae-1aae-4e8e-b12f-d3727346ae51\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") "
Oct 30 21:01:35.676919 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.676771    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-podinfo\") pod \"675355ae-1aae-4e8e-b12f-d3727346ae51\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") "
Oct 30 21:01:35.676919 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.676828    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-data" pod "675355ae-1aae-4e8e-b12f-d3727346ae51" (UID: "675355ae-1aae-4e8e-b12f-d3727346ae51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:35.676919 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.676841    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-envoy\") pod \"675355ae-1aae-4e8e-b12f-d3727346ae51\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") "
Oct 30 21:01:35.677084 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.676944    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-token\" (UniqueName: \"kubernetes.io/projected/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-token\") pod \"675355ae-1aae-4e8e-b12f-d3727346ae51\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") "
Oct 30 21:01:35.677084 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.676986    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-workload-certs\") pod \"675355ae-1aae-4e8e-b12f-d3727346ae51\" (UID: \"675355ae-1aae-4e8e-b12f-d3727346ae51\") "
Oct 30 21:01:35.677084 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.676992    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-workload-socket" pod "675355ae-1aae-4e8e-b12f-d3727346ae51" (UID: "675355ae-1aae-4e8e-b12f-d3727346ae51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:35.677084 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.677060    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-credential-socket" pod "675355ae-1aae-4e8e-b12f-d3727346ae51" (UID: "675355ae-1aae-4e8e-b12f-d3727346ae51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:35.677303 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.677123    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/675355ae-1aae-4e8e-b12f-d3727346ae51-istiod-ca-cert" pod "675355ae-1aae-4e8e-b12f-d3727346ae51" (UID: "675355ae-1aae-4e8e-b12f-d3727346ae51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:35.677385 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.677369    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-workload-certs" pod "675355ae-1aae-4e8e-b12f-d3727346ae51" (UID: "675355ae-1aae-4e8e-b12f-d3727346ae51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:35.677493 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.677472    4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-workload-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:35.677722 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.677498    4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-data\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:35.677722 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.677512    4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/675355ae-1aae-4e8e-b12f-d3727346ae51-istiod-ca-cert\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:35.677722 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.677526    4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-workload-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:35.677722 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.677540    4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-credential-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:35.679215 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.679192    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-envoy" pod "675355ae-1aae-4e8e-b12f-d3727346ae51" (UID: "675355ae-1aae-4e8e-b12f-d3727346ae51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:01:35.679342 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.679272    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-token" pod "675355ae-1aae-4e8e-b12f-d3727346ae51" (UID: "675355ae-1aae-4e8e-b12f-d3727346ae51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:35.679679 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.679660    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-podinfo" pod "675355ae-1aae-4e8e-b12f-d3727346ae51" (UID: "675355ae-1aae-4e8e-b12f-d3727346ae51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:01:35.679679 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.679666    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675355ae-1aae-4e8e-b12f-d3727346ae51-kube-api-access-rmw2j" pod "675355ae-1aae-4e8e-b12f-d3727346ae51" (UID: "675355ae-1aae-4e8e-b12f-d3727346ae51"). InnerVolumeSpecName "kube-api-access-rmw2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:35.778550 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.778519    4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-podinfo\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:35.778550 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.778544    4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-envoy\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:35.778550 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.778554    4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/675355ae-1aae-4e8e-b12f-d3727346ae51-istio-token\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:35.778767 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.778565    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmw2j\" (UniqueName: \"kubernetes.io/projected/675355ae-1aae-4e8e-b12f-d3727346ae51-kube-api-access-rmw2j\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:35.986200 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.986163    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"]
Oct 30 21:01:35.988946 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:35.988925    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5dc13e-openshift-default-69d59fb4dc-9thmb"]
Oct 30 21:01:36.431138 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:01:36.431107    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="675355ae-1aae-4e8e-b12f-d3727346ae51" path="/var/lib/kubelet/pods/675355ae-1aae-4e8e-b12f-d3727346ae51/volumes"
Oct 30 21:02:00.954283 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:02:00.954244    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:02:30.954510 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:02:30.954431    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:03:00.954276 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:03:00.954238    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:03:20.443982 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:20.443943    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"]
Oct 30 21:03:30.954575 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:03:30.954533    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:03:35.464091 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:35.464037    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524" podUID="31cc010d-9e1f-42c2-a736-0ceacbedb8c9" containerName="main" containerID="cri-o://205eec1bf53a4b6425a8710fc9ee8bc97cf1a836d0798654d7518170baf1e69f" gracePeriod=45
Oct 30 21:03:35.610454 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:35.610279    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524" podUID="31cc010d-9e1f-42c2-a736-0ceacbedb8c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": dial tcp 10.133.0.37:8000: connect: connection refused"
Oct 30 21:03:35.720131 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:35.720079    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 21:03:35.872747 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:35.872711    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-tls-certs\") pod \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") "
Oct 30 21:03:35.872909 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:35.872844    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-tmp-dir\") pod \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") "
Oct 30 21:03:35.872988 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:35.872970    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-dshm\") pod \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") "
Oct 30 21:03:35.873046 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:35.873028    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-model-cache\") pod \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") "
Oct 30 21:03:35.873089 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:35.873061    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-home\" (UniqueName: \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-home\") pod \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") "
Oct 30 21:03:35.873144 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:35.873089    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-kube-api-access-m7tvc\" (UniqueName: \"kubernetes.io/projected/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-kube-api-access-m7tvc\") pod \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\" (UID: \"31cc010d-9e1f-42c2-a736-0ceacbedb8c9\") "
Oct 30 21:03:35.873144 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:35.873088    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-tmp-dir" pod "31cc010d-9e1f-42c2-a736-0ceacbedb8c9" (UID: "31cc010d-9e1f-42c2-a736-0ceacbedb8c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:35.873372 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:35.873341    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-model-cache" pod "31cc010d-9e1f-42c2-a736-0ceacbedb8c9" (UID: "31cc010d-9e1f-42c2-a736-0ceacbedb8c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:35.873477 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:35.873349    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-home" pod "31cc010d-9e1f-42c2-a736-0ceacbedb8c9" (UID: "31cc010d-9e1f-42c2-a736-0ceacbedb8c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:35.873575 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:35.873559    4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-tmp-dir\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:35.873632 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:35.873582    4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-model-cache\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:35.873632 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:35.873598    4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-home\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:35.875551 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:35.875517    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-kube-api-access-m7tvc" pod "31cc010d-9e1f-42c2-a736-0ceacbedb8c9" (UID: "31cc010d-9e1f-42c2-a736-0ceacbedb8c9"). InnerVolumeSpecName "kube-api-access-m7tvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:35.875659 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:35.875574    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-dshm" pod "31cc010d-9e1f-42c2-a736-0ceacbedb8c9" (UID: "31cc010d-9e1f-42c2-a736-0ceacbedb8c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:35.875659 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:35.875577    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-tls-certs" pod "31cc010d-9e1f-42c2-a736-0ceacbedb8c9" (UID: "31cc010d-9e1f-42c2-a736-0ceacbedb8c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:35.974638 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:35.974554    4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-dshm\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:35.974638 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:35.974584    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7tvc\" (UniqueName: \"kubernetes.io/projected/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-kube-api-access-m7tvc\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:35.974638 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:35.974594    4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31cc010d-9e1f-42c2-a736-0ceacbedb8c9-tls-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:36.203688 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:36.203652    4471 generic.go:356] "Generic (PLEG): container finished" podID="31cc010d-9e1f-42c2-a736-0ceacbedb8c9" containerID="205eec1bf53a4b6425a8710fc9ee8bc97cf1a836d0798654d7518170baf1e69f" exitCode=0
Oct 30 21:03:36.203958 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:36.203704    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"
Oct 30 21:03:36.203958 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:36.203701    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524" event={"ID":"31cc010d-9e1f-42c2-a736-0ceacbedb8c9","Type":"ContainerDied","Data":"205eec1bf53a4b6425a8710fc9ee8bc97cf1a836d0798654d7518170baf1e69f"}
Oct 30 21:03:36.203958 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:36.203816    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524" event={"ID":"31cc010d-9e1f-42c2-a736-0ceacbedb8c9","Type":"ContainerDied","Data":"4ec6d1c989e350c829f6cc6c18ba2ad68043e2f67db6349ca21c52a5539b1236"}
Oct 30 21:03:36.203958 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:36.203841    4471 scope.go:122] "RemoveContainer" containerID="205eec1bf53a4b6425a8710fc9ee8bc97cf1a836d0798654d7518170baf1e69f"
Oct 30 21:03:36.217152 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:36.217096    4471 scope.go:122] "RemoveContainer" containerID="205eec1bf53a4b6425a8710fc9ee8bc97cf1a836d0798654d7518170baf1e69f"
Oct 30 21:03:36.217538 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:03:36.217510    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"205eec1bf53a4b6425a8710fc9ee8bc97cf1a836d0798654d7518170baf1e69f\": container with ID starting with 205eec1bf53a4b6425a8710fc9ee8bc97cf1a836d0798654d7518170baf1e69f not found: ID does not exist" containerID="205eec1bf53a4b6425a8710fc9ee8bc97cf1a836d0798654d7518170baf1e69f"
Oct 30 21:03:36.217621 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:36.217552    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"205eec1bf53a4b6425a8710fc9ee8bc97cf1a836d0798654d7518170baf1e69f"} err="failed to get container status \"205eec1bf53a4b6425a8710fc9ee8bc97cf1a836d0798654d7518170baf1e69f\": rpc error: code = NotFound desc = could not find container \"205eec1bf53a4b6425a8710fc9ee8bc97cf1a836d0798654d7518170baf1e69f\": container with ID starting with 205eec1bf53a4b6425a8710fc9ee8bc97cf1a836d0798654d7518170baf1e69f not found: ID does not exist"
Oct 30 21:03:36.365595 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:36.365507    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"]
Oct 30 21:03:36.367397 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:36.367373    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-df031b/simulator-w2-df031b-kserve-7d847db4b5-4l524"]
Oct 30 21:03:36.435956 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:36.435920    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31cc010d-9e1f-42c2-a736-0ceacbedb8c9" path="/var/lib/kubelet/pods/31cc010d-9e1f-42c2-a736-0ceacbedb8c9/volumes"
Oct 30 21:03:45.944629 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:45.944596    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"]
Oct 30 21:03:45.994463 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:45.994417    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"]
Oct 30 21:03:45.996690 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:03:45.996658    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"]
Oct 30 21:04:00.955297 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:04:00.955216    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:04:00.965158 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:00.965128    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl" podUID="a4344406-c869-420f-8cf3-93c5e0549be8" containerName="main" containerID="cri-o://778b52228592ae82ecfc42f9b9fe3498024af28f8f7f9b397167c5fc40634657" gracePeriod=45
Oct 30 21:04:01.015986 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.015928    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf" podUID="bf89b9b8-0419-48bc-b410-12807dab75db" containerName="main" containerID="cri-o://864dd860845797f85fd55c5d481611ef736050a92557acb65fc07b87404709ad" gracePeriod=45
Oct 30 21:04:01.017183 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.017139    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4" podUID="0ec28e09-b926-4e73-8f40-2f1ab3a41df8" containerName="main" containerID="cri-o://ef2bbee2f623f68975ea752df53ea4fc676723306f18add0cfda14046d8fc268" gracePeriod=45
Oct 30 21:04:01.115049 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.115008    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf" podUID="bf89b9b8-0419-48bc-b410-12807dab75db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": dial tcp 10.133.0.45:8000: connect: connection refused"
Oct 30 21:04:01.206920 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.206614    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4" podUID="0ec28e09-b926-4e73-8f40-2f1ab3a41df8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": dial tcp 10.133.0.39:8000: connect: connection refused"
Oct 30 21:04:01.291813 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.291786    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 21:04:01.314041 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.314014    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 21:04:01.322870 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.322849    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 21:04:01.343113 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.343086    4471 generic.go:356] "Generic (PLEG): container finished" podID="0ec28e09-b926-4e73-8f40-2f1ab3a41df8" containerID="ef2bbee2f623f68975ea752df53ea4fc676723306f18add0cfda14046d8fc268" exitCode=0
Oct 30 21:04:01.343267 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.343189    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4" event={"ID":"0ec28e09-b926-4e73-8f40-2f1ab3a41df8","Type":"ContainerDied","Data":"ef2bbee2f623f68975ea752df53ea4fc676723306f18add0cfda14046d8fc268"}
Oct 30 21:04:01.343267 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.343225    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"
Oct 30 21:04:01.343267 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.343239    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4" event={"ID":"0ec28e09-b926-4e73-8f40-2f1ab3a41df8","Type":"ContainerDied","Data":"933e3fcebe8c2f395fdaee75972d53362b225462bfd4a7883bf98df6cf96289b"}
Oct 30 21:04:01.343267 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.343259    4471 scope.go:122] "RemoveContainer" containerID="ef2bbee2f623f68975ea752df53ea4fc676723306f18add0cfda14046d8fc268"
Oct 30 21:04:01.344521 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.344500    4471 generic.go:356] "Generic (PLEG): container finished" podID="a4344406-c869-420f-8cf3-93c5e0549be8" containerID="778b52228592ae82ecfc42f9b9fe3498024af28f8f7f9b397167c5fc40634657" exitCode=0
Oct 30 21:04:01.344647 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.344528    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl" event={"ID":"a4344406-c869-420f-8cf3-93c5e0549be8","Type":"ContainerDied","Data":"778b52228592ae82ecfc42f9b9fe3498024af28f8f7f9b397167c5fc40634657"}
Oct 30 21:04:01.344647 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.344563    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl" event={"ID":"a4344406-c869-420f-8cf3-93c5e0549be8","Type":"ContainerDied","Data":"b7f899b94d9be5d752907f9038bcc16a326d38f763cab308f1b1b533a85cd0dd"}
Oct 30 21:04:01.344647 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.344538    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"
Oct 30 21:04:01.345907 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.345884    4471 generic.go:356] "Generic (PLEG): container finished" podID="bf89b9b8-0419-48bc-b410-12807dab75db" containerID="864dd860845797f85fd55c5d481611ef736050a92557acb65fc07b87404709ad" exitCode=0
Oct 30 21:04:01.345990 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.345918    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"
Oct 30 21:04:01.345990 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.345964    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf" event={"ID":"bf89b9b8-0419-48bc-b410-12807dab75db","Type":"ContainerDied","Data":"864dd860845797f85fd55c5d481611ef736050a92557acb65fc07b87404709ad"}
Oct 30 21:04:01.346074 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.345995    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf" event={"ID":"bf89b9b8-0419-48bc-b410-12807dab75db","Type":"ContainerDied","Data":"fda68d39ee39e0044e48921699ddd611d53dec2cf6c496179f5e6e8e76eeff4a"}
Oct 30 21:04:01.351184 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.351164    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4344406-c869-420f-8cf3-93c5e0549be8-kube-api-access-g4glb\" (UniqueName: \"kubernetes.io/projected/a4344406-c869-420f-8cf3-93c5e0549be8-kube-api-access-g4glb\") pod \"a4344406-c869-420f-8cf3-93c5e0549be8\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") "
Oct 30 21:04:01.351272 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.351207    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-tmp-dir\") pod \"a4344406-c869-420f-8cf3-93c5e0549be8\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") "
Oct 30 21:04:01.351272 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.351268    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-home\" (UniqueName: \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-home\") pod \"a4344406-c869-420f-8cf3-93c5e0549be8\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") "
Oct 30 21:04:01.351760 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.351300    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4344406-c869-420f-8cf3-93c5e0549be8-tls-certs\" (UniqueName: \"kubernetes.io/secret/a4344406-c869-420f-8cf3-93c5e0549be8-tls-certs\") pod \"a4344406-c869-420f-8cf3-93c5e0549be8\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") "
Oct 30 21:04:01.351760 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.351350    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-model-cache\") pod \"a4344406-c869-420f-8cf3-93c5e0549be8\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") "
Oct 30 21:04:01.351760 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.351508    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-dshm\") pod \"a4344406-c869-420f-8cf3-93c5e0549be8\" (UID: \"a4344406-c869-420f-8cf3-93c5e0549be8\") "
Oct 30 21:04:01.351760 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.351572    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-tmp-dir" pod "a4344406-c869-420f-8cf3-93c5e0549be8" (UID: "a4344406-c869-420f-8cf3-93c5e0549be8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:01.351760 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.351604    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-model-cache" pod "a4344406-c869-420f-8cf3-93c5e0549be8" (UID: "a4344406-c869-420f-8cf3-93c5e0549be8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:01.351760 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.351627    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-home" pod "a4344406-c869-420f-8cf3-93c5e0549be8" (UID: "a4344406-c869-420f-8cf3-93c5e0549be8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:01.352129 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.352080    4471 scope.go:122] "RemoveContainer" containerID="ef2bbee2f623f68975ea752df53ea4fc676723306f18add0cfda14046d8fc268"
Oct 30 21:04:01.352187 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.352147    4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-home\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.352225 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.352185    4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-model-cache\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.352225 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.352201    4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-tmp-dir\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.352406 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:04:01.352380    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef2bbee2f623f68975ea752df53ea4fc676723306f18add0cfda14046d8fc268\": container with ID starting with ef2bbee2f623f68975ea752df53ea4fc676723306f18add0cfda14046d8fc268 not found: ID does not exist" containerID="ef2bbee2f623f68975ea752df53ea4fc676723306f18add0cfda14046d8fc268"
Oct 30 21:04:01.352460 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.352420    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef2bbee2f623f68975ea752df53ea4fc676723306f18add0cfda14046d8fc268"} err="failed to get container status \"ef2bbee2f623f68975ea752df53ea4fc676723306f18add0cfda14046d8fc268\": rpc error: code = NotFound desc = could not find container \"ef2bbee2f623f68975ea752df53ea4fc676723306f18add0cfda14046d8fc268\": container with ID starting with ef2bbee2f623f68975ea752df53ea4fc676723306f18add0cfda14046d8fc268 not found: ID does not exist"
Oct 30 21:04:01.352460 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.352449    4471 scope.go:122] "RemoveContainer" containerID="778b52228592ae82ecfc42f9b9fe3498024af28f8f7f9b397167c5fc40634657"
Oct 30 21:04:01.353815 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.353786    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4344406-c869-420f-8cf3-93c5e0549be8-tls-certs" pod "a4344406-c869-420f-8cf3-93c5e0549be8" (UID: "a4344406-c869-420f-8cf3-93c5e0549be8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:04:01.353815 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.353802    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4344406-c869-420f-8cf3-93c5e0549be8-kube-api-access-g4glb" pod "a4344406-c869-420f-8cf3-93c5e0549be8" (UID: "a4344406-c869-420f-8cf3-93c5e0549be8"). InnerVolumeSpecName "kube-api-access-g4glb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:01.353997 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.353982    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-dshm" pod "a4344406-c869-420f-8cf3-93c5e0549be8" (UID: "a4344406-c869-420f-8cf3-93c5e0549be8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:01.360602 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.360581    4471 scope.go:122] "RemoveContainer" containerID="778b52228592ae82ecfc42f9b9fe3498024af28f8f7f9b397167c5fc40634657"
Oct 30 21:04:01.360852 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:04:01.360836    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"778b52228592ae82ecfc42f9b9fe3498024af28f8f7f9b397167c5fc40634657\": container with ID starting with 778b52228592ae82ecfc42f9b9fe3498024af28f8f7f9b397167c5fc40634657 not found: ID does not exist" containerID="778b52228592ae82ecfc42f9b9fe3498024af28f8f7f9b397167c5fc40634657"
Oct 30 21:04:01.360914 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.360859    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"778b52228592ae82ecfc42f9b9fe3498024af28f8f7f9b397167c5fc40634657"} err="failed to get container status \"778b52228592ae82ecfc42f9b9fe3498024af28f8f7f9b397167c5fc40634657\": rpc error: code = NotFound desc = could not find container \"778b52228592ae82ecfc42f9b9fe3498024af28f8f7f9b397167c5fc40634657\": container with ID starting with 778b52228592ae82ecfc42f9b9fe3498024af28f8f7f9b397167c5fc40634657 not found: ID does not exist"
Oct 30 21:04:01.360914 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.360874    4471 scope.go:122] "RemoveContainer" containerID="864dd860845797f85fd55c5d481611ef736050a92557acb65fc07b87404709ad"
Oct 30 21:04:01.368837 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.368817    4471 scope.go:122] "RemoveContainer" containerID="864dd860845797f85fd55c5d481611ef736050a92557acb65fc07b87404709ad"
Oct 30 21:04:01.369103 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:04:01.369085    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"864dd860845797f85fd55c5d481611ef736050a92557acb65fc07b87404709ad\": container with ID starting with 864dd860845797f85fd55c5d481611ef736050a92557acb65fc07b87404709ad not found: ID does not exist" containerID="864dd860845797f85fd55c5d481611ef736050a92557acb65fc07b87404709ad"
Oct 30 21:04:01.369172 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.369110    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"864dd860845797f85fd55c5d481611ef736050a92557acb65fc07b87404709ad"} err="failed to get container status \"864dd860845797f85fd55c5d481611ef736050a92557acb65fc07b87404709ad\": rpc error: code = NotFound desc = could not find container \"864dd860845797f85fd55c5d481611ef736050a92557acb65fc07b87404709ad\": container with ID starting with 864dd860845797f85fd55c5d481611ef736050a92557acb65fc07b87404709ad not found: ID does not exist"
Oct 30 21:04:01.453184 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.453135    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-home\" (UniqueName: \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-home\") pod \"bf89b9b8-0419-48bc-b410-12807dab75db\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") "
Oct 30 21:04:01.453387 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.453212    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-tmp-dir\") pod \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") "
Oct 30 21:04:01.453387 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.453252    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf89b9b8-0419-48bc-b410-12807dab75db-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf89b9b8-0419-48bc-b410-12807dab75db-tls-certs\") pod \"bf89b9b8-0419-48bc-b410-12807dab75db\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") "
Oct 30 21:04:01.453387 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.453295    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-tmp-dir\") pod \"bf89b9b8-0419-48bc-b410-12807dab75db\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") "
Oct 30 21:04:01.453387 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.453349    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-model-cache\") pod \"bf89b9b8-0419-48bc-b410-12807dab75db\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") "
Oct 30 21:04:01.453728 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.453399    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-dshm\") pod \"bf89b9b8-0419-48bc-b410-12807dab75db\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") "
Oct 30 21:04:01.453728 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.453439    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-model-cache\") pod \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") "
Oct 30 21:04:01.453728 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.453492    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-dshm\") pod \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") "
Oct 30 21:04:01.453728 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.453493    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-tmp-dir" pod "0ec28e09-b926-4e73-8f40-2f1ab3a41df8" (UID: "0ec28e09-b926-4e73-8f40-2f1ab3a41df8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:01.453728 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.453589    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-tls-certs\") pod \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") "
Oct 30 21:04:01.453728 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.453648    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-home\" (UniqueName: \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-home\") pod \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") "
Oct 30 21:04:01.453728 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.453688    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-model-cache" pod "bf89b9b8-0419-48bc-b410-12807dab75db" (UID: "bf89b9b8-0419-48bc-b410-12807dab75db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:01.454086 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.453734    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-kube-api-access-4wnt7\" (UniqueName: \"kubernetes.io/projected/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-kube-api-access-4wnt7\") pod \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\" (UID: \"0ec28e09-b926-4e73-8f40-2f1ab3a41df8\") "
Oct 30 21:04:01.454086 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.453781    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf89b9b8-0419-48bc-b410-12807dab75db-kube-api-access-cznfl\" (UniqueName: \"kubernetes.io/projected/bf89b9b8-0419-48bc-b410-12807dab75db-kube-api-access-cznfl\") pod \"bf89b9b8-0419-48bc-b410-12807dab75db\" (UID: \"bf89b9b8-0419-48bc-b410-12807dab75db\") "
Oct 30 21:04:01.454086 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.453861    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-tmp-dir" pod "bf89b9b8-0419-48bc-b410-12807dab75db" (UID: "bf89b9b8-0419-48bc-b410-12807dab75db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:01.454246 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.454140    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4glb\" (UniqueName: \"kubernetes.io/projected/a4344406-c869-420f-8cf3-93c5e0549be8-kube-api-access-g4glb\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.454246 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.454160    4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-tmp-dir\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.454246 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.454175    4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-tmp-dir\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.454246 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.454189    4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-model-cache\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.454246 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.454205    4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4344406-c869-420f-8cf3-93c5e0549be8-tls-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.454246 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.454219    4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4344406-c869-420f-8cf3-93c5e0549be8-dshm\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.454562 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.454423    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-home" pod "bf89b9b8-0419-48bc-b410-12807dab75db" (UID: "bf89b9b8-0419-48bc-b410-12807dab75db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:01.454805 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.454780    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-model-cache" pod "0ec28e09-b926-4e73-8f40-2f1ab3a41df8" (UID: "0ec28e09-b926-4e73-8f40-2f1ab3a41df8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:01.455113 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.455088    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-home" pod "0ec28e09-b926-4e73-8f40-2f1ab3a41df8" (UID: "0ec28e09-b926-4e73-8f40-2f1ab3a41df8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:01.457418 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.457300    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-tls-certs" pod "0ec28e09-b926-4e73-8f40-2f1ab3a41df8" (UID: "0ec28e09-b926-4e73-8f40-2f1ab3a41df8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:04:01.457556 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.457479    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-dshm" pod "bf89b9b8-0419-48bc-b410-12807dab75db" (UID: "bf89b9b8-0419-48bc-b410-12807dab75db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:01.457719 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.457695    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf89b9b8-0419-48bc-b410-12807dab75db-tls-certs" pod "bf89b9b8-0419-48bc-b410-12807dab75db" (UID: "bf89b9b8-0419-48bc-b410-12807dab75db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:04:01.458253 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.458217    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-dshm" pod "0ec28e09-b926-4e73-8f40-2f1ab3a41df8" (UID: "0ec28e09-b926-4e73-8f40-2f1ab3a41df8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:01.458253 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.458239    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf89b9b8-0419-48bc-b410-12807dab75db-kube-api-access-cznfl" pod "bf89b9b8-0419-48bc-b410-12807dab75db" (UID: "bf89b9b8-0419-48bc-b410-12807dab75db"). InnerVolumeSpecName "kube-api-access-cznfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:01.459160 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.459135    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-kube-api-access-4wnt7" pod "0ec28e09-b926-4e73-8f40-2f1ab3a41df8" (UID: "0ec28e09-b926-4e73-8f40-2f1ab3a41df8"). InnerVolumeSpecName "kube-api-access-4wnt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:01.555165 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.555126    4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-tls-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.555165 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.555152    4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-home\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.555165 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.555164    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wnt7\" (UniqueName: \"kubernetes.io/projected/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-kube-api-access-4wnt7\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.555425 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.555174    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cznfl\" (UniqueName: \"kubernetes.io/projected/bf89b9b8-0419-48bc-b410-12807dab75db-kube-api-access-cznfl\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.555425 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.555184    4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-home\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.555425 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.555192    4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf89b9b8-0419-48bc-b410-12807dab75db-tls-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.555425 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.555199    4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf89b9b8-0419-48bc-b410-12807dab75db-dshm\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.555425 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.555207    4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-model-cache\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.555425 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.555216    4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ec28e09-b926-4e73-8f40-2f1ab3a41df8-dshm\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:01.798922 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.798833    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"]
Oct 30 21:04:01.800684 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.800663    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/distinct-2-w3-ae2d38-kserve-59669cc6bb-b4tgl"]
Oct 30 21:04:01.907695 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.907668    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"]
Oct 30 21:04:01.911274 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:01.911249    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/distinct-w3-ae2d38-kserve-84564db7fc-979bf"]
Oct 30 21:04:02.019023 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:02.018990    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"]
Oct 30 21:04:02.022422 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:02.022397    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ae2d38/premium-w3-ae2d38-kserve-6577db47cd-45ns4"]
Oct 30 21:04:02.431095 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:02.431062    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ec28e09-b926-4e73-8f40-2f1ab3a41df8" path="/var/lib/kubelet/pods/0ec28e09-b926-4e73-8f40-2f1ab3a41df8/volumes"
Oct 30 21:04:02.431505 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:02.431491    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4344406-c869-420f-8cf3-93c5e0549be8" path="/var/lib/kubelet/pods/a4344406-c869-420f-8cf3-93c5e0549be8/volumes"
Oct 30 21:04:02.431849 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:02.431837    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf89b9b8-0419-48bc-b410-12807dab75db" path="/var/lib/kubelet/pods/bf89b9b8-0419-48bc-b410-12807dab75db/volumes"
Oct 30 21:04:15.091795 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:15.091760    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"]
Oct 30 21:04:27.595211 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:27.593826    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"]
Oct 30 21:04:27.601395 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:27.599529    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276" podUID="ccae7148-e9b3-488b-ad71-acceff9f3d9a" containerName="maas-api" containerID="cri-o://dd2ab414012358b8895b10913188ab28724ea3c51841a4e8a8241e1c2e36ca24" gracePeriod=30
Oct 30 21:04:27.916972 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:27.916934    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"
Oct 30 21:04:28.021678 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:28.021651    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccae7148-e9b3-488b-ad71-acceff9f3d9a-kube-api-access-rvpzn\" (UniqueName: \"kubernetes.io/projected/ccae7148-e9b3-488b-ad71-acceff9f3d9a-kube-api-access-rvpzn\") pod \"ccae7148-e9b3-488b-ad71-acceff9f3d9a\" (UID: \"ccae7148-e9b3-488b-ad71-acceff9f3d9a\") "
Oct 30 21:04:28.021859 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:28.021697    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccae7148-e9b3-488b-ad71-acceff9f3d9a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccae7148-e9b3-488b-ad71-acceff9f3d9a-openshift-service-ca\") pod \"ccae7148-e9b3-488b-ad71-acceff9f3d9a\" (UID: \"ccae7148-e9b3-488b-ad71-acceff9f3d9a\") "
Oct 30 21:04:28.021859 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:28.021721    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccae7148-e9b3-488b-ad71-acceff9f3d9a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccae7148-e9b3-488b-ad71-acceff9f3d9a-maas-api-tls\") pod \"ccae7148-e9b3-488b-ad71-acceff9f3d9a\" (UID: \"ccae7148-e9b3-488b-ad71-acceff9f3d9a\") "
Oct 30 21:04:28.021859 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:28.021737    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccae7148-e9b3-488b-ad71-acceff9f3d9a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccae7148-e9b3-488b-ad71-acceff9f3d9a-metrics-certs\") pod \"ccae7148-e9b3-488b-ad71-acceff9f3d9a\" (UID: \"ccae7148-e9b3-488b-ad71-acceff9f3d9a\") "
Oct 30 21:04:28.022088 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:28.022059    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccae7148-e9b3-488b-ad71-acceff9f3d9a-openshift-service-ca" pod "ccae7148-e9b3-488b-ad71-acceff9f3d9a" (UID: "ccae7148-e9b3-488b-ad71-acceff9f3d9a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:04:28.024030 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:28.024003    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccae7148-e9b3-488b-ad71-acceff9f3d9a-metrics-certs" pod "ccae7148-e9b3-488b-ad71-acceff9f3d9a" (UID: "ccae7148-e9b3-488b-ad71-acceff9f3d9a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:04:28.024145 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:28.024028    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccae7148-e9b3-488b-ad71-acceff9f3d9a-maas-api-tls" pod "ccae7148-e9b3-488b-ad71-acceff9f3d9a" (UID: "ccae7148-e9b3-488b-ad71-acceff9f3d9a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:04:28.024145 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:28.024082    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccae7148-e9b3-488b-ad71-acceff9f3d9a-kube-api-access-rvpzn" pod "ccae7148-e9b3-488b-ad71-acceff9f3d9a" (UID: "ccae7148-e9b3-488b-ad71-acceff9f3d9a"). InnerVolumeSpecName "kube-api-access-rvpzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:28.122864 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:28.122829    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvpzn\" (UniqueName: \"kubernetes.io/projected/ccae7148-e9b3-488b-ad71-acceff9f3d9a-kube-api-access-rvpzn\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:28.122864 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:28.122865    4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccae7148-e9b3-488b-ad71-acceff9f3d9a-openshift-service-ca\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:28.123066 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:28.122882    4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccae7148-e9b3-488b-ad71-acceff9f3d9a-maas-api-tls\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:28.123066 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:28.122897    4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccae7148-e9b3-488b-ad71-acceff9f3d9a-metrics-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:28.520638 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:28.520595    4471 generic.go:356] "Generic (PLEG): container finished" podID="ccae7148-e9b3-488b-ad71-acceff9f3d9a" containerID="dd2ab414012358b8895b10913188ab28724ea3c51841a4e8a8241e1c2e36ca24" exitCode=0
Oct 30 21:04:28.520780 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:28.520649    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"
Oct 30 21:04:28.520780 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:28.520695    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276" event={"ID":"ccae7148-e9b3-488b-ad71-acceff9f3d9a","Type":"ContainerDied","Data":"dd2ab414012358b8895b10913188ab28724ea3c51841a4e8a8241e1c2e36ca24"}
Oct 30 21:04:28.520780 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:28.520756    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276" event={"ID":"ccae7148-e9b3-488b-ad71-acceff9f3d9a","Type":"ContainerDied","Data":"601b5641fd6666bd712a76226312673f673be8ba8052351b6db5632de861ab10"}
Oct 30 21:04:28.520938 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:28.520784    4471 scope.go:122] "RemoveContainer" containerID="dd2ab414012358b8895b10913188ab28724ea3c51841a4e8a8241e1c2e36ca24"
Oct 30 21:04:28.531708 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:28.531689    4471 scope.go:122] "RemoveContainer" containerID="dd2ab414012358b8895b10913188ab28724ea3c51841a4e8a8241e1c2e36ca24"
Oct 30 21:04:28.532007 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:04:28.531985    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd2ab414012358b8895b10913188ab28724ea3c51841a4e8a8241e1c2e36ca24\": container with ID starting with dd2ab414012358b8895b10913188ab28724ea3c51841a4e8a8241e1c2e36ca24 not found: ID does not exist" containerID="dd2ab414012358b8895b10913188ab28724ea3c51841a4e8a8241e1c2e36ca24"
Oct 30 21:04:28.532070 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:28.532017    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd2ab414012358b8895b10913188ab28724ea3c51841a4e8a8241e1c2e36ca24"} err="failed to get container status \"dd2ab414012358b8895b10913188ab28724ea3c51841a4e8a8241e1c2e36ca24\": rpc error: code = NotFound desc = could not find container \"dd2ab414012358b8895b10913188ab28724ea3c51841a4e8a8241e1c2e36ca24\": container with ID starting with dd2ab414012358b8895b10913188ab28724ea3c51841a4e8a8241e1c2e36ca24 not found: ID does not exist"
Oct 30 21:04:28.683136 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:28.683101    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"]
Oct 30 21:04:28.686158 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:28.686135    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-df031b-6787d94787-2m276"]
Oct 30 21:04:30.111965 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.111924    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws" podUID="096c8245-e0ed-4aa9-ad32-7aca48ab6aa4" containerName="main" containerID="cri-o://e64f57d0c6fd80b2aad196732196d0cbc19ef1379b7357e29492722eac82bd27" gracePeriod=45
Oct 30 21:04:30.167177 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.167138    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws" podUID="096c8245-e0ed-4aa9-ad32-7aca48ab6aa4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": dial tcp 10.133.0.98:8000: connect: connection refused"
Oct 30 21:04:30.364251 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.364193    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:04:30.430586 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.430554    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccae7148-e9b3-488b-ad71-acceff9f3d9a" path="/var/lib/kubelet/pods/ccae7148-e9b3-488b-ad71-acceff9f3d9a/volumes"
Oct 30 21:04:30.464669 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.464637    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-tmp-dir\") pod \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") "
Oct 30 21:04:30.464833 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.464724    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-home\" (UniqueName: \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-home\") pod \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") "
Oct 30 21:04:30.464833 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.464760    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-dshm\") pod \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") "
Oct 30 21:04:30.464833 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.464801    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-model-cache\") pod \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") "
Oct 30 21:04:30.464833 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.464822    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-kube-api-access-tkcb2\" (UniqueName: \"kubernetes.io/projected/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-kube-api-access-tkcb2\") pod \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") "
Oct 30 21:04:30.465061 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.464883    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-tls-certs\" (UniqueName: \"kubernetes.io/secret/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-tls-certs\") pod \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\" (UID: \"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4\") "
Oct 30 21:04:30.465061 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.464897    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-tmp-dir" pod "096c8245-e0ed-4aa9-ad32-7aca48ab6aa4" (UID: "096c8245-e0ed-4aa9-ad32-7aca48ab6aa4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:30.465061 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.465047    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-model-cache" pod "096c8245-e0ed-4aa9-ad32-7aca48ab6aa4" (UID: "096c8245-e0ed-4aa9-ad32-7aca48ab6aa4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:30.465212 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.465065    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-home" pod "096c8245-e0ed-4aa9-ad32-7aca48ab6aa4" (UID: "096c8245-e0ed-4aa9-ad32-7aca48ab6aa4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:30.465212 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.465199    4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-home\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:30.465285 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.465217    4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-model-cache\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:30.465285 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.465255    4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-tmp-dir\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:30.467233 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.467204    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-dshm" pod "096c8245-e0ed-4aa9-ad32-7aca48ab6aa4" (UID: "096c8245-e0ed-4aa9-ad32-7aca48ab6aa4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:30.467377 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.467229    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-tls-certs" pod "096c8245-e0ed-4aa9-ad32-7aca48ab6aa4" (UID: "096c8245-e0ed-4aa9-ad32-7aca48ab6aa4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:04:30.467377 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.467304    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-kube-api-access-tkcb2" pod "096c8245-e0ed-4aa9-ad32-7aca48ab6aa4" (UID: "096c8245-e0ed-4aa9-ad32-7aca48ab6aa4"). InnerVolumeSpecName "kube-api-access-tkcb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:30.535005 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.534974    4471 generic.go:356] "Generic (PLEG): container finished" podID="096c8245-e0ed-4aa9-ad32-7aca48ab6aa4" containerID="e64f57d0c6fd80b2aad196732196d0cbc19ef1379b7357e29492722eac82bd27" exitCode=0
Oct 30 21:04:30.535178 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.535018    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"
Oct 30 21:04:30.535178 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.535056    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws" event={"ID":"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4","Type":"ContainerDied","Data":"e64f57d0c6fd80b2aad196732196d0cbc19ef1379b7357e29492722eac82bd27"}
Oct 30 21:04:30.535178 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.535097    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws" event={"ID":"096c8245-e0ed-4aa9-ad32-7aca48ab6aa4","Type":"ContainerDied","Data":"35dddfd3acc99012d1eda0260249f5aa16855948020890c2f8f241f96c5d681f"}
Oct 30 21:04:30.535178 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.535112    4471 scope.go:122] "RemoveContainer" containerID="e64f57d0c6fd80b2aad196732196d0cbc19ef1379b7357e29492722eac82bd27"
Oct 30 21:04:30.544240 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.544218    4471 scope.go:122] "RemoveContainer" containerID="e64f57d0c6fd80b2aad196732196d0cbc19ef1379b7357e29492722eac82bd27"
Oct 30 21:04:30.544544 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:04:30.544525    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e64f57d0c6fd80b2aad196732196d0cbc19ef1379b7357e29492722eac82bd27\": container with ID starting with e64f57d0c6fd80b2aad196732196d0cbc19ef1379b7357e29492722eac82bd27 not found: ID does not exist" containerID="e64f57d0c6fd80b2aad196732196d0cbc19ef1379b7357e29492722eac82bd27"
Oct 30 21:04:30.544598 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.544554    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e64f57d0c6fd80b2aad196732196d0cbc19ef1379b7357e29492722eac82bd27"} err="failed to get container status \"e64f57d0c6fd80b2aad196732196d0cbc19ef1379b7357e29492722eac82bd27\": rpc error: code = NotFound desc = could not find container \"e64f57d0c6fd80b2aad196732196d0cbc19ef1379b7357e29492722eac82bd27\": container with ID starting with e64f57d0c6fd80b2aad196732196d0cbc19ef1379b7357e29492722eac82bd27 not found: ID does not exist"
Oct 30 21:04:30.566127 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.566095    4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-tls-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:30.566127 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.566122    4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-dshm\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:30.566275 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.566136    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkcb2\" (UniqueName: \"kubernetes.io/projected/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4-kube-api-access-tkcb2\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:30.699192 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.699158    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"]
Oct 30 21:04:30.702203 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:30.702175    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-11a0b0/test-model-11a0b0-kserve-d94dd4cbf-b2pws"]
Oct 30 21:04:30.954656 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:04:30.954574    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:04:32.432245 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:32.432209    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="096c8245-e0ed-4aa9-ad32-7aca48ab6aa4" path="/var/lib/kubelet/pods/096c8245-e0ed-4aa9-ad32-7aca48ab6aa4/volumes"
Oct 30 21:04:57.545366 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:57.545329    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"]
Oct 30 21:04:57.545902 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:57.545551    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk" podUID="c5652acc-b0ac-4755-a655-407ae5fba9a3" containerName="maas-api" containerID="cri-o://52b604c5f43b91311c9012b5e23276c3a59fc2c29fa5e691c4a7573ac030acc8" gracePeriod=30
Oct 30 21:04:57.841662 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:57.841635    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"
Oct 30 21:04:57.970361 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:57.970305    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5652acc-b0ac-4755-a655-407ae5fba9a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5652acc-b0ac-4755-a655-407ae5fba9a3-metrics-certs\") pod \"c5652acc-b0ac-4755-a655-407ae5fba9a3\" (UID: \"c5652acc-b0ac-4755-a655-407ae5fba9a3\") "
Oct 30 21:04:57.970527 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:57.970411    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5652acc-b0ac-4755-a655-407ae5fba9a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5652acc-b0ac-4755-a655-407ae5fba9a3-openshift-service-ca\") pod \"c5652acc-b0ac-4755-a655-407ae5fba9a3\" (UID: \"c5652acc-b0ac-4755-a655-407ae5fba9a3\") "
Oct 30 21:04:57.970527 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:57.970452    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5652acc-b0ac-4755-a655-407ae5fba9a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5652acc-b0ac-4755-a655-407ae5fba9a3-maas-api-tls\") pod \"c5652acc-b0ac-4755-a655-407ae5fba9a3\" (UID: \"c5652acc-b0ac-4755-a655-407ae5fba9a3\") "
Oct 30 21:04:57.970527 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:57.970486    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5652acc-b0ac-4755-a655-407ae5fba9a3-kube-api-access-rrmtv\" (UniqueName: \"kubernetes.io/projected/c5652acc-b0ac-4755-a655-407ae5fba9a3-kube-api-access-rrmtv\") pod \"c5652acc-b0ac-4755-a655-407ae5fba9a3\" (UID: \"c5652acc-b0ac-4755-a655-407ae5fba9a3\") "
Oct 30 21:04:57.970838 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:57.970814    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5652acc-b0ac-4755-a655-407ae5fba9a3-openshift-service-ca" pod "c5652acc-b0ac-4755-a655-407ae5fba9a3" (UID: "c5652acc-b0ac-4755-a655-407ae5fba9a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:04:57.973311 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:57.973279    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5652acc-b0ac-4755-a655-407ae5fba9a3-metrics-certs" pod "c5652acc-b0ac-4755-a655-407ae5fba9a3" (UID: "c5652acc-b0ac-4755-a655-407ae5fba9a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:04:57.973471 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:57.973348    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5652acc-b0ac-4755-a655-407ae5fba9a3-kube-api-access-rrmtv" pod "c5652acc-b0ac-4755-a655-407ae5fba9a3" (UID: "c5652acc-b0ac-4755-a655-407ae5fba9a3"). InnerVolumeSpecName "kube-api-access-rrmtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:57.974793 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:57.974768    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5652acc-b0ac-4755-a655-407ae5fba9a3-maas-api-tls" pod "c5652acc-b0ac-4755-a655-407ae5fba9a3" (UID: "c5652acc-b0ac-4755-a655-407ae5fba9a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:04:58.074079 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.071825    4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5652acc-b0ac-4755-a655-407ae5fba9a3-metrics-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:58.074079 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.071867    4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5652acc-b0ac-4755-a655-407ae5fba9a3-openshift-service-ca\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:58.074079 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.071884    4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5652acc-b0ac-4755-a655-407ae5fba9a3-maas-api-tls\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:58.074079 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.071899    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrmtv\" (UniqueName: \"kubernetes.io/projected/c5652acc-b0ac-4755-a655-407ae5fba9a3-kube-api-access-rrmtv\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:58.665705 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.665664    4471 generic.go:356] "Generic (PLEG): container finished" podID="c5652acc-b0ac-4755-a655-407ae5fba9a3" containerID="52b604c5f43b91311c9012b5e23276c3a59fc2c29fa5e691c4a7573ac030acc8" exitCode=0
Oct 30 21:04:58.666244 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.665723    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"
Oct 30 21:04:58.666244 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.665743    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk" event={"ID":"c5652acc-b0ac-4755-a655-407ae5fba9a3","Type":"ContainerDied","Data":"52b604c5f43b91311c9012b5e23276c3a59fc2c29fa5e691c4a7573ac030acc8"}
Oct 30 21:04:58.666244 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.665807    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk" event={"ID":"c5652acc-b0ac-4755-a655-407ae5fba9a3","Type":"ContainerDied","Data":"830f1b599d59257c5c0f8486d5f45b3d7cd7938025dba3dbcf0375e12b83eead"}
Oct 30 21:04:58.666244 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.665837    4471 scope.go:122] "RemoveContainer" containerID="52b604c5f43b91311c9012b5e23276c3a59fc2c29fa5e691c4a7573ac030acc8"
Oct 30 21:04:58.666920 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.666876    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"]
Oct 30 21:04:58.667088 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.667073    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv" podUID="811787ae-bcf5-452e-9c13-2ada1d001705" containerName="maas-api" containerID="cri-o://042b697cc31f6527ba5389c602f49385ccb73ee3d45c276c1633c8e807c43c6b" gracePeriod=30
Oct 30 21:04:58.679135 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.679111    4471 scope.go:122] "RemoveContainer" containerID="52b604c5f43b91311c9012b5e23276c3a59fc2c29fa5e691c4a7573ac030acc8"
Oct 30 21:04:58.679528 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:04:58.679501    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52b604c5f43b91311c9012b5e23276c3a59fc2c29fa5e691c4a7573ac030acc8\": container with ID starting with 52b604c5f43b91311c9012b5e23276c3a59fc2c29fa5e691c4a7573ac030acc8 not found: ID does not exist" containerID="52b604c5f43b91311c9012b5e23276c3a59fc2c29fa5e691c4a7573ac030acc8"
Oct 30 21:04:58.679633 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.679541    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b604c5f43b91311c9012b5e23276c3a59fc2c29fa5e691c4a7573ac030acc8"} err="failed to get container status \"52b604c5f43b91311c9012b5e23276c3a59fc2c29fa5e691c4a7573ac030acc8\": rpc error: code = NotFound desc = could not find container \"52b604c5f43b91311c9012b5e23276c3a59fc2c29fa5e691c4a7573ac030acc8\": container with ID starting with 52b604c5f43b91311c9012b5e23276c3a59fc2c29fa5e691c4a7573ac030acc8 not found: ID does not exist"
Oct 30 21:04:58.829094 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.829024    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"]
Oct 30 21:04:58.835139 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.835107    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ae2d38-76f7566fbf-6whwk"]
Oct 30 21:04:58.933180 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.933154    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"
Oct 30 21:04:58.980774 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.980732    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/811787ae-bcf5-452e-9c13-2ada1d001705-metrics-certs\" (UniqueName: \"kubernetes.io/secret/811787ae-bcf5-452e-9c13-2ada1d001705-metrics-certs\") pod \"811787ae-bcf5-452e-9c13-2ada1d001705\" (UID: \"811787ae-bcf5-452e-9c13-2ada1d001705\") "
Oct 30 21:04:58.980983 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.980841    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/811787ae-bcf5-452e-9c13-2ada1d001705-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/811787ae-bcf5-452e-9c13-2ada1d001705-maas-api-tls\") pod \"811787ae-bcf5-452e-9c13-2ada1d001705\" (UID: \"811787ae-bcf5-452e-9c13-2ada1d001705\") "
Oct 30 21:04:58.980983 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.980904    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/811787ae-bcf5-452e-9c13-2ada1d001705-kube-api-access-t4mjr\" (UniqueName: \"kubernetes.io/projected/811787ae-bcf5-452e-9c13-2ada1d001705-kube-api-access-t4mjr\") pod \"811787ae-bcf5-452e-9c13-2ada1d001705\" (UID: \"811787ae-bcf5-452e-9c13-2ada1d001705\") "
Oct 30 21:04:58.980983 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.980945    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/811787ae-bcf5-452e-9c13-2ada1d001705-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/811787ae-bcf5-452e-9c13-2ada1d001705-openshift-service-ca\") pod \"811787ae-bcf5-452e-9c13-2ada1d001705\" (UID: \"811787ae-bcf5-452e-9c13-2ada1d001705\") "
Oct 30 21:04:58.981566 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.981537    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811787ae-bcf5-452e-9c13-2ada1d001705-openshift-service-ca" pod "811787ae-bcf5-452e-9c13-2ada1d001705" (UID: "811787ae-bcf5-452e-9c13-2ada1d001705"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:04:58.983116 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.983088    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/811787ae-bcf5-452e-9c13-2ada1d001705-maas-api-tls" pod "811787ae-bcf5-452e-9c13-2ada1d001705" (UID: "811787ae-bcf5-452e-9c13-2ada1d001705"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:04:58.983266 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.983243    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811787ae-bcf5-452e-9c13-2ada1d001705-kube-api-access-t4mjr" pod "811787ae-bcf5-452e-9c13-2ada1d001705" (UID: "811787ae-bcf5-452e-9c13-2ada1d001705"). InnerVolumeSpecName "kube-api-access-t4mjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:58.983436 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:58.983417    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/811787ae-bcf5-452e-9c13-2ada1d001705-metrics-certs" pod "811787ae-bcf5-452e-9c13-2ada1d001705" (UID: "811787ae-bcf5-452e-9c13-2ada1d001705"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:04:59.085038 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:59.084949    4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/811787ae-bcf5-452e-9c13-2ada1d001705-maas-api-tls\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:59.085285 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:59.085263    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4mjr\" (UniqueName: \"kubernetes.io/projected/811787ae-bcf5-452e-9c13-2ada1d001705-kube-api-access-t4mjr\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:59.085480 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:59.085420    4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/811787ae-bcf5-452e-9c13-2ada1d001705-openshift-service-ca\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:59.085595 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:59.085582    4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/811787ae-bcf5-452e-9c13-2ada1d001705-metrics-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:59.672378 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:59.672346    4471 generic.go:356] "Generic (PLEG): container finished" podID="811787ae-bcf5-452e-9c13-2ada1d001705" containerID="042b697cc31f6527ba5389c602f49385ccb73ee3d45c276c1633c8e807c43c6b" exitCode=0
Oct 30 21:04:59.672800 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:59.672394    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"
Oct 30 21:04:59.672800 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:59.672436    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv" event={"ID":"811787ae-bcf5-452e-9c13-2ada1d001705","Type":"ContainerDied","Data":"042b697cc31f6527ba5389c602f49385ccb73ee3d45c276c1633c8e807c43c6b"}
Oct 30 21:04:59.672800 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:59.672492    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv" event={"ID":"811787ae-bcf5-452e-9c13-2ada1d001705","Type":"ContainerDied","Data":"672939958a032a82e236e03ff11f80c6b2f2c637bb25d1869c17e34292d2d471"}
Oct 30 21:04:59.672800 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:59.672520    4471 scope.go:122] "RemoveContainer" containerID="042b697cc31f6527ba5389c602f49385ccb73ee3d45c276c1633c8e807c43c6b"
Oct 30 21:04:59.689421 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:59.689401    4471 scope.go:122] "RemoveContainer" containerID="042b697cc31f6527ba5389c602f49385ccb73ee3d45c276c1633c8e807c43c6b"
Oct 30 21:04:59.689743 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:04:59.689724    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042b697cc31f6527ba5389c602f49385ccb73ee3d45c276c1633c8e807c43c6b\": container with ID starting with 042b697cc31f6527ba5389c602f49385ccb73ee3d45c276c1633c8e807c43c6b not found: ID does not exist" containerID="042b697cc31f6527ba5389c602f49385ccb73ee3d45c276c1633c8e807c43c6b"
Oct 30 21:04:59.689792 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:59.689752    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042b697cc31f6527ba5389c602f49385ccb73ee3d45c276c1633c8e807c43c6b"} err="failed to get container status \"042b697cc31f6527ba5389c602f49385ccb73ee3d45c276c1633c8e807c43c6b\": rpc error: code = NotFound desc = could not find container \"042b697cc31f6527ba5389c602f49385ccb73ee3d45c276c1633c8e807c43c6b\": container with ID starting with 042b697cc31f6527ba5389c602f49385ccb73ee3d45c276c1633c8e807c43c6b not found: ID does not exist"
Oct 30 21:04:59.835621 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:59.835583    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"]
Oct 30 21:04:59.884820 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:04:59.884788    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bcef0e-589d47d487-md9bv"]
Oct 30 21:05:00.430242 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:00.430207    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="811787ae-bcf5-452e-9c13-2ada1d001705" path="/var/lib/kubelet/pods/811787ae-bcf5-452e-9c13-2ada1d001705/volumes"
Oct 30 21:05:00.430640 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:00.430626    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5652acc-b0ac-4755-a655-407ae5fba9a3" path="/var/lib/kubelet/pods/c5652acc-b0ac-4755-a655-407ae5fba9a3/volumes"
Oct 30 21:05:00.954701 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:05:00.954667    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:05:06.801461 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:06.801425    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"]
Oct 30 21:05:06.801960 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:06.801641    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw" podUID="7b68692e-aa7c-4b66-a316-19e71881d64e" containerName="istio-proxy" containerID="cri-o://e58f8fcb15afa6e447fcdbb5bfe44e25693cdca99d1c1a953090e15fe27740cb" gracePeriod=30
Oct 30 21:05:12.061784 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.061758    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 21:05:12.127178 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.127147    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-data\") pod \"7b68692e-aa7c-4b66-a316-19e71881d64e\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") "
Oct 30 21:05:12.127314 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.127242    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b68692e-aa7c-4b66-a316-19e71881d64e-kube-api-access-49dws\" (UniqueName: \"kubernetes.io/projected/7b68692e-aa7c-4b66-a316-19e71881d64e-kube-api-access-49dws\") pod \"7b68692e-aa7c-4b66-a316-19e71881d64e\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") "
Oct 30 21:05:12.127314 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.127268    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-envoy\") pod \"7b68692e-aa7c-4b66-a316-19e71881d64e\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") "
Oct 30 21:05:12.127314 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.127287    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-podinfo\") pod \"7b68692e-aa7c-4b66-a316-19e71881d64e\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") "
Oct 30 21:05:12.127482 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.127342    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b68692e-aa7c-4b66-a316-19e71881d64e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b68692e-aa7c-4b66-a316-19e71881d64e-istiod-ca-cert\") pod \"7b68692e-aa7c-4b66-a316-19e71881d64e\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") "
Oct 30 21:05:12.127482 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.127393    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-workload-certs\") pod \"7b68692e-aa7c-4b66-a316-19e71881d64e\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") "
Oct 30 21:05:12.127482 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.127421    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-token\" (UniqueName: \"kubernetes.io/projected/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-token\") pod \"7b68692e-aa7c-4b66-a316-19e71881d64e\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") "
Oct 30 21:05:12.127482 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.127448    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-workload-socket\") pod \"7b68692e-aa7c-4b66-a316-19e71881d64e\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") "
Oct 30 21:05:12.127482 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.127451    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-data" pod "7b68692e-aa7c-4b66-a316-19e71881d64e" (UID: "7b68692e-aa7c-4b66-a316-19e71881d64e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:12.127482 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.127478    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-credential-socket\") pod \"7b68692e-aa7c-4b66-a316-19e71881d64e\" (UID: \"7b68692e-aa7c-4b66-a316-19e71881d64e\") "
Oct 30 21:05:12.127861 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.127829    4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-data\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:12.128461 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.128422    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-workload-socket" pod "7b68692e-aa7c-4b66-a316-19e71881d64e" (UID: "7b68692e-aa7c-4b66-a316-19e71881d64e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:12.128556 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.128454    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-workload-certs" pod "7b68692e-aa7c-4b66-a316-19e71881d64e" (UID: "7b68692e-aa7c-4b66-a316-19e71881d64e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:12.128556 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.128524    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b68692e-aa7c-4b66-a316-19e71881d64e-istiod-ca-cert" pod "7b68692e-aa7c-4b66-a316-19e71881d64e" (UID: "7b68692e-aa7c-4b66-a316-19e71881d64e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:05:12.128683 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.128647    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-credential-socket" pod "7b68692e-aa7c-4b66-a316-19e71881d64e" (UID: "7b68692e-aa7c-4b66-a316-19e71881d64e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:12.129748 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.129722    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-envoy" pod "7b68692e-aa7c-4b66-a316-19e71881d64e" (UID: "7b68692e-aa7c-4b66-a316-19e71881d64e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:12.129826 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.129799    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-podinfo" pod "7b68692e-aa7c-4b66-a316-19e71881d64e" (UID: "7b68692e-aa7c-4b66-a316-19e71881d64e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:05:12.129876 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.129846    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b68692e-aa7c-4b66-a316-19e71881d64e-kube-api-access-49dws" pod "7b68692e-aa7c-4b66-a316-19e71881d64e" (UID: "7b68692e-aa7c-4b66-a316-19e71881d64e"). InnerVolumeSpecName "kube-api-access-49dws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:12.130524 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.130509    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-token" pod "7b68692e-aa7c-4b66-a316-19e71881d64e" (UID: "7b68692e-aa7c-4b66-a316-19e71881d64e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:12.229253 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.229223    4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-workload-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:12.229534 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.229509    4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-token\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:12.229647 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.229537    4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-workload-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:12.229647 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.229553    4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-credential-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:12.229647 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.229567    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49dws\" (UniqueName: \"kubernetes.io/projected/7b68692e-aa7c-4b66-a316-19e71881d64e-kube-api-access-49dws\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:12.229647 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.229581    4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-envoy\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:12.229647 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.229590    4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b68692e-aa7c-4b66-a316-19e71881d64e-istio-podinfo\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:12.229647 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.229599    4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b68692e-aa7c-4b66-a316-19e71881d64e-istiod-ca-cert\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:12.735960 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.735860    4471 generic.go:356] "Generic (PLEG): container finished" podID="7b68692e-aa7c-4b66-a316-19e71881d64e" containerID="e58f8fcb15afa6e447fcdbb5bfe44e25693cdca99d1c1a953090e15fe27740cb" exitCode=0
Oct 30 21:05:12.735960 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.735909    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"
Oct 30 21:05:12.735960 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.735948    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw" event={"ID":"7b68692e-aa7c-4b66-a316-19e71881d64e","Type":"ContainerDied","Data":"e58f8fcb15afa6e447fcdbb5bfe44e25693cdca99d1c1a953090e15fe27740cb"}
Oct 30 21:05:12.736221 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.736004    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw" event={"ID":"7b68692e-aa7c-4b66-a316-19e71881d64e","Type":"ContainerDied","Data":"24ab28322e8be1c38d00802f70d50303281e9caf349b6e9f4aa09153e7fd532d"}
Oct 30 21:05:12.736221 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.736025    4471 scope.go:122] "RemoveContainer" containerID="e58f8fcb15afa6e447fcdbb5bfe44e25693cdca99d1c1a953090e15fe27740cb"
Oct 30 21:05:12.746675 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.746652    4471 scope.go:122] "RemoveContainer" containerID="e58f8fcb15afa6e447fcdbb5bfe44e25693cdca99d1c1a953090e15fe27740cb"
Oct 30 21:05:12.746956 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:05:12.746938    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e58f8fcb15afa6e447fcdbb5bfe44e25693cdca99d1c1a953090e15fe27740cb\": container with ID starting with e58f8fcb15afa6e447fcdbb5bfe44e25693cdca99d1c1a953090e15fe27740cb not found: ID does not exist" containerID="e58f8fcb15afa6e447fcdbb5bfe44e25693cdca99d1c1a953090e15fe27740cb"
Oct 30 21:05:12.747015 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.746964    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e58f8fcb15afa6e447fcdbb5bfe44e25693cdca99d1c1a953090e15fe27740cb"} err="failed to get container status \"e58f8fcb15afa6e447fcdbb5bfe44e25693cdca99d1c1a953090e15fe27740cb\": rpc error: code = NotFound desc = could not find container \"e58f8fcb15afa6e447fcdbb5bfe44e25693cdca99d1c1a953090e15fe27740cb\": container with ID starting with e58f8fcb15afa6e447fcdbb5bfe44e25693cdca99d1c1a953090e15fe27740cb not found: ID does not exist"
Oct 30 21:05:12.897958 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.897922    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"]
Oct 30 21:05:12.947552 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:12.947489    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ae2d38-openshift-default-7bfb7d67c-z5tdw"]
Oct 30 21:05:14.430617 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:14.430578    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b68692e-aa7c-4b66-a316-19e71881d64e" path="/var/lib/kubelet/pods/7b68692e-aa7c-4b66-a316-19e71881d64e/volumes"
Oct 30 21:05:30.954295 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:05:30.954209    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:05:48.343582 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:48.343547    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"]
Oct 30 21:05:48.393869 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:48.393829    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"]
Oct 30 21:05:48.395437 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:05:48.395412    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"]
Oct 30 21:06:00.957018 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:06:00.956979    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:06:03.363896 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.363834    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk" podUID="a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98" containerName="main" containerID="cri-o://7feb2c5b2fa335e2980d845e6cb99260508f72fbf9d10653dc974917ae5accc1" gracePeriod=45
Oct 30 21:06:03.414763 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.414714    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht" podUID="5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" containerName="main" containerID="cri-o://43f5dce5f0b4055f71c0ca1f00de1b629d5374465f813a13ef67d5ec52f107b9" gracePeriod=45
Oct 30 21:06:03.415944 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.415893    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b" podUID="1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf" containerName="main" containerID="cri-o://c5944b541c6f90b78ae628e8fb2bc04667b6d6264eac9a32a71486d53b378522" gracePeriod=45
Oct 30 21:06:03.521855 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.521710    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht" podUID="5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": dial tcp 10.133.0.41:8000: connect: connection refused"
Oct 30 21:06:03.684000 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.683974    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 21:06:03.688180 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.688154    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 21:06:03.713710 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.713684    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 21:06:03.749896 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.749867    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-dshm\") pod \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") "
Oct 30 21:06:03.750049 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.749906    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-home\" (UniqueName: \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-home\") pod \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") "
Oct 30 21:06:03.750049 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.749942    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-kube-api-access-hfg5q\" (UniqueName: \"kubernetes.io/projected/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-kube-api-access-hfg5q\") pod \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") "
Oct 30 21:06:03.750049 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.749970    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-tmp-dir\") pod \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") "
Oct 30 21:06:03.750049 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.750015    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-model-cache\") pod \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") "
Oct 30 21:06:03.750269 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.750054    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-dshm\") pod \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") "
Oct 30 21:06:03.750269 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.750082    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-model-cache\") pod \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") "
Oct 30 21:06:03.750269 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.750141    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-tls-certs\") pod \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") "
Oct 30 21:06:03.750269 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.750183    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-kube-api-access-45j64\" (UniqueName: \"kubernetes.io/projected/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-kube-api-access-45j64\") pod \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") "
Oct 30 21:06:03.750269 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.750190    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-home" pod "a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98" (UID: "a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:03.750269 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.750231    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-tmp-dir\") pod \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") "
Oct 30 21:06:03.750590 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.750281    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-model-cache" pod "1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf" (UID: "1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:03.750590 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.750308    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-tls-certs\") pod \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\" (UID: \"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98\") "
Oct 30 21:06:03.750590 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.750404    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-home\" (UniqueName: \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-home\") pod \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\" (UID: \"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf\") "
Oct 30 21:06:03.750590 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.750434    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-model-cache" pod "a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98" (UID: "a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:03.750590 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.750569    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-tmp-dir" pod "a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98" (UID: "a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:03.750590 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.750578    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-tmp-dir" pod "1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf" (UID: "1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:03.750877 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.750738    4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-home\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.750877 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.750756    4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-tmp-dir\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.750877 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.750772    4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-model-cache\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.750877 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.750787    4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-model-cache\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.750877 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.750801    4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-tmp-dir\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.750877 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.750821    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-home" pod "1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf" (UID: "1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:03.752766 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.752740    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-tls-certs" pod "a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98" (UID: "a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:06:03.752887 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.752842    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-dshm" pod "a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98" (UID: "a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:03.752970 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.752953    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-tls-certs" pod "1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf" (UID: "1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:06:03.752970 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.752960    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-dshm" pod "1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf" (UID: "1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:03.753087 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.752958    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-kube-api-access-hfg5q" pod "a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98" (UID: "a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98"). InnerVolumeSpecName "kube-api-access-hfg5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:03.753153 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.753139    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-kube-api-access-45j64" pod "1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf" (UID: "1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf"). InnerVolumeSpecName "kube-api-access-45j64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:03.851452 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.851408    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-dshm\") pod \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") "
Oct 30 21:06:03.851638 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.851471    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-home\" (UniqueName: \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-home\") pod \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") "
Oct 30 21:06:03.851638 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.851512    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-tmp-dir\") pod \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") "
Oct 30 21:06:03.851638 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.851557    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-tls-certs\") pod \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") "
Oct 30 21:06:03.851638 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.851594    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-kube-api-access-7ffv6\" (UniqueName: \"kubernetes.io/projected/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-kube-api-access-7ffv6\") pod \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") "
Oct 30 21:06:03.851638 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.851635    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-model-cache\") pod \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\" (UID: \"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea\") "
Oct 30 21:06:03.851916 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.851800    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-home" pod "5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" (UID: "5bf5e57f-4c83-4c4b-9771-5f78d48d53ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:03.852001 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.851980    4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-home\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.852065 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.852006    4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-home\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.852065 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.852015    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-model-cache" pod "5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" (UID: "5bf5e57f-4c83-4c4b-9771-5f78d48d53ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:03.852065 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.852020    4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-dshm\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.852432 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.852074    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfg5q\" (UniqueName: \"kubernetes.io/projected/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-kube-api-access-hfg5q\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.852432 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.852092    4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-dshm\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.852432 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.852108    4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-tls-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.852432 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.852123    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45j64\" (UniqueName: \"kubernetes.io/projected/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf-kube-api-access-45j64\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.852432 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.852143    4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98-tls-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.852432 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.852419    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-tmp-dir" pod "5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" (UID: "5bf5e57f-4c83-4c4b-9771-5f78d48d53ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:03.853872 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.853847    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-dshm" pod "5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" (UID: "5bf5e57f-4c83-4c4b-9771-5f78d48d53ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:03.854627 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.854598    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-tls-certs" pod "5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" (UID: "5bf5e57f-4c83-4c4b-9771-5f78d48d53ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:06:03.854627 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.854616    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-kube-api-access-7ffv6" pod "5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" (UID: "5bf5e57f-4c83-4c4b-9771-5f78d48d53ea"). InnerVolumeSpecName "kube-api-access-7ffv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:03.953484 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.953398    4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-model-cache\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.953484 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.953426    4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-dshm\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.953484 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.953436    4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-tmp-dir\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.953484 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.953444    4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-tls-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.953484 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.953455    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ffv6\" (UniqueName: \"kubernetes.io/projected/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea-kube-api-access-7ffv6\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:03.970447 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.970416    4471 generic.go:356] "Generic (PLEG): container finished" podID="5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" containerID="43f5dce5f0b4055f71c0ca1f00de1b629d5374465f813a13ef67d5ec52f107b9" exitCode=0
Oct 30 21:06:03.970589 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.970485    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"
Oct 30 21:06:03.970589 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.970500    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht" event={"ID":"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea","Type":"ContainerDied","Data":"43f5dce5f0b4055f71c0ca1f00de1b629d5374465f813a13ef67d5ec52f107b9"}
Oct 30 21:06:03.970589 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.970548    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht" event={"ID":"5bf5e57f-4c83-4c4b-9771-5f78d48d53ea","Type":"ContainerDied","Data":"5c8a17a2e4a3416b9a1df395838abd696c8a4ae16c6c7719075a59eb32e9baaf"}
Oct 30 21:06:03.970589 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.970572    4471 scope.go:122] "RemoveContainer" containerID="43f5dce5f0b4055f71c0ca1f00de1b629d5374465f813a13ef67d5ec52f107b9"
Oct 30 21:06:03.971918 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.971870    4471 generic.go:356] "Generic (PLEG): container finished" podID="a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98" containerID="7feb2c5b2fa335e2980d845e6cb99260508f72fbf9d10653dc974917ae5accc1" exitCode=0
Oct 30 21:06:03.971918 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.971918    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"
Oct 30 21:06:03.972117 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.971963    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk" event={"ID":"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98","Type":"ContainerDied","Data":"7feb2c5b2fa335e2980d845e6cb99260508f72fbf9d10653dc974917ae5accc1"}
Oct 30 21:06:03.972117 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.972040    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk" event={"ID":"a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98","Type":"ContainerDied","Data":"ffa66152675dc7d3a3a9bbc80c069b6c03df71de3e51d34531c165b0eb3bdda9"}
Oct 30 21:06:03.973660 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.973635    4471 generic.go:356] "Generic (PLEG): container finished" podID="1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf" containerID="c5944b541c6f90b78ae628e8fb2bc04667b6d6264eac9a32a71486d53b378522" exitCode=0
Oct 30 21:06:03.973799 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.973695    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"
Oct 30 21:06:03.973799 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.973730    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b" event={"ID":"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf","Type":"ContainerDied","Data":"c5944b541c6f90b78ae628e8fb2bc04667b6d6264eac9a32a71486d53b378522"}
Oct 30 21:06:03.973799 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.973768    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b" event={"ID":"1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf","Type":"ContainerDied","Data":"e169789f3765034cd1fd126545f77455f0a548179164157fb4288595dae01b66"}
Oct 30 21:06:03.982474 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.982452    4471 scope.go:122] "RemoveContainer" containerID="43f5dce5f0b4055f71c0ca1f00de1b629d5374465f813a13ef67d5ec52f107b9"
Oct 30 21:06:03.982845 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:06:03.982822    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43f5dce5f0b4055f71c0ca1f00de1b629d5374465f813a13ef67d5ec52f107b9\": container with ID starting with 43f5dce5f0b4055f71c0ca1f00de1b629d5374465f813a13ef67d5ec52f107b9 not found: ID does not exist" containerID="43f5dce5f0b4055f71c0ca1f00de1b629d5374465f813a13ef67d5ec52f107b9"
Oct 30 21:06:03.982939 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.982851    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43f5dce5f0b4055f71c0ca1f00de1b629d5374465f813a13ef67d5ec52f107b9"} err="failed to get container status \"43f5dce5f0b4055f71c0ca1f00de1b629d5374465f813a13ef67d5ec52f107b9\": rpc error: code = NotFound desc = could not find container \"43f5dce5f0b4055f71c0ca1f00de1b629d5374465f813a13ef67d5ec52f107b9\": container with ID starting with 43f5dce5f0b4055f71c0ca1f00de1b629d5374465f813a13ef67d5ec52f107b9 not found: ID does not exist"
Oct 30 21:06:03.982939 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.982869    4471 scope.go:122] "RemoveContainer" containerID="7feb2c5b2fa335e2980d845e6cb99260508f72fbf9d10653dc974917ae5accc1"
Oct 30 21:06:03.991787 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.991766    4471 scope.go:122] "RemoveContainer" containerID="7feb2c5b2fa335e2980d845e6cb99260508f72fbf9d10653dc974917ae5accc1"
Oct 30 21:06:03.992064 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:06:03.992047    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7feb2c5b2fa335e2980d845e6cb99260508f72fbf9d10653dc974917ae5accc1\": container with ID starting with 7feb2c5b2fa335e2980d845e6cb99260508f72fbf9d10653dc974917ae5accc1 not found: ID does not exist" containerID="7feb2c5b2fa335e2980d845e6cb99260508f72fbf9d10653dc974917ae5accc1"
Oct 30 21:06:03.992117 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.992071    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7feb2c5b2fa335e2980d845e6cb99260508f72fbf9d10653dc974917ae5accc1"} err="failed to get container status \"7feb2c5b2fa335e2980d845e6cb99260508f72fbf9d10653dc974917ae5accc1\": rpc error: code = NotFound desc = could not find container \"7feb2c5b2fa335e2980d845e6cb99260508f72fbf9d10653dc974917ae5accc1\": container with ID starting with 7feb2c5b2fa335e2980d845e6cb99260508f72fbf9d10653dc974917ae5accc1 not found: ID does not exist"
Oct 30 21:06:03.992117 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.992089    4471 scope.go:122] "RemoveContainer" containerID="c5944b541c6f90b78ae628e8fb2bc04667b6d6264eac9a32a71486d53b378522"
Oct 30 21:06:03.999399 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.999379    4471 scope.go:122] "RemoveContainer" containerID="c5944b541c6f90b78ae628e8fb2bc04667b6d6264eac9a32a71486d53b378522"
Oct 30 21:06:03.999665 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:06:03.999643    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5944b541c6f90b78ae628e8fb2bc04667b6d6264eac9a32a71486d53b378522\": container with ID starting with c5944b541c6f90b78ae628e8fb2bc04667b6d6264eac9a32a71486d53b378522 not found: ID does not exist" containerID="c5944b541c6f90b78ae628e8fb2bc04667b6d6264eac9a32a71486d53b378522"
Oct 30 21:06:03.999733 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:03.999674    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5944b541c6f90b78ae628e8fb2bc04667b6d6264eac9a32a71486d53b378522"} err="failed to get container status \"c5944b541c6f90b78ae628e8fb2bc04667b6d6264eac9a32a71486d53b378522\": rpc error: code = NotFound desc = could not find container \"c5944b541c6f90b78ae628e8fb2bc04667b6d6264eac9a32a71486d53b378522\": container with ID starting with c5944b541c6f90b78ae628e8fb2bc04667b6d6264eac9a32a71486d53b378522 not found: ID does not exist"
Oct 30 21:06:04.139357 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:04.139261    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"]
Oct 30 21:06:04.142867 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:04.142838    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/distinct-w0-0a78ac-kserve-cd57bfc89-4p4ht"]
Oct 30 21:06:04.249465 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:04.249370    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"]
Oct 30 21:06:04.252897 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:04.252870    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/distinct-2-w0-0a78ac-kserve-558b9b874-78zdk"]
Oct 30 21:06:04.361045 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:04.361010    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"]
Oct 30 21:06:04.365189 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:04.365162    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0a78ac/simulator-w0-0a78ac-kserve-859c9547ff-m5f8b"]
Oct 30 21:06:04.429810 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:04.429776    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf" path="/var/lib/kubelet/pods/1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf/volumes"
Oct 30 21:06:04.430274 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:04.430254    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" path="/var/lib/kubelet/pods/5bf5e57f-4c83-4c4b-9771-5f78d48d53ea/volumes"
Oct 30 21:06:04.430721 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:06:04.430700    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98" path="/var/lib/kubelet/pods/a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98/volumes"
Oct 30 21:06:30.953698 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:06:30.953658    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:07:00.953046 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:07:00.952953    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:07:30.955626 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:07:30.955500    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:08:00.954047 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:08:00.954009    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:08:30.953620 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:08:30.953537    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:09:00.953202 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:09:00.953165    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:09:22.123019 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:22.122984    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"]
Oct 30 21:09:30.953557 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:09:30.953510    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:09:37.143873 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.143838    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46" podUID="13896eb7-f9cc-4cf2-9364-b723651037af" containerName="main" containerID="cri-o://a6e63d232f90d0c0755e5c950af33ed9ff0a5aa077dc4d75976d435640860ab9" gracePeriod=45
Oct 30 21:09:37.388581 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.388558    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 21:09:37.478082 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.478013    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13896eb7-f9cc-4cf2-9364-b723651037af-tls-certs\" (UniqueName: \"kubernetes.io/secret/13896eb7-f9cc-4cf2-9364-b723651037af-tls-certs\") pod \"13896eb7-f9cc-4cf2-9364-b723651037af\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") "
Oct 30 21:09:37.478082 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.478074    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-tmp-dir\") pod \"13896eb7-f9cc-4cf2-9364-b723651037af\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") "
Oct 30 21:09:37.478265 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.478097    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-model-cache\") pod \"13896eb7-f9cc-4cf2-9364-b723651037af\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") "
Oct 30 21:09:37.478265 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.478172    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-dshm\") pod \"13896eb7-f9cc-4cf2-9364-b723651037af\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") "
Oct 30 21:09:37.478265 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.478197    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-home\" (UniqueName: \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-home\") pod \"13896eb7-f9cc-4cf2-9364-b723651037af\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") "
Oct 30 21:09:37.478265 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.478228    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13896eb7-f9cc-4cf2-9364-b723651037af-kube-api-access-ztp59\" (UniqueName: \"kubernetes.io/projected/13896eb7-f9cc-4cf2-9364-b723651037af-kube-api-access-ztp59\") pod \"13896eb7-f9cc-4cf2-9364-b723651037af\" (UID: \"13896eb7-f9cc-4cf2-9364-b723651037af\") "
Oct 30 21:09:37.478814 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.478308    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-tmp-dir" pod "13896eb7-f9cc-4cf2-9364-b723651037af" (UID: "13896eb7-f9cc-4cf2-9364-b723651037af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:37.478814 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.478497    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-model-cache" pod "13896eb7-f9cc-4cf2-9364-b723651037af" (UID: "13896eb7-f9cc-4cf2-9364-b723651037af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:37.478814 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.478507    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-home" pod "13896eb7-f9cc-4cf2-9364-b723651037af" (UID: "13896eb7-f9cc-4cf2-9364-b723651037af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:37.478814 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.478632    4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-home\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.478814 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.478650    4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-tmp-dir\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.478814 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.478666    4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-model-cache\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.480853 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.480808    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-dshm" pod "13896eb7-f9cc-4cf2-9364-b723651037af" (UID: "13896eb7-f9cc-4cf2-9364-b723651037af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:09:37.481083 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.480936    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13896eb7-f9cc-4cf2-9364-b723651037af-kube-api-access-ztp59" pod "13896eb7-f9cc-4cf2-9364-b723651037af" (UID: "13896eb7-f9cc-4cf2-9364-b723651037af"). InnerVolumeSpecName "kube-api-access-ztp59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:37.481384 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.481364    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13896eb7-f9cc-4cf2-9364-b723651037af-tls-certs" pod "13896eb7-f9cc-4cf2-9364-b723651037af" (UID: "13896eb7-f9cc-4cf2-9364-b723651037af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:09:37.579822 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.579787    4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13896eb7-f9cc-4cf2-9364-b723651037af-dshm\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.579822 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.579815    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztp59\" (UniqueName: \"kubernetes.io/projected/13896eb7-f9cc-4cf2-9364-b723651037af-kube-api-access-ztp59\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.579822 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.579829    4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13896eb7-f9cc-4cf2-9364-b723651037af-tls-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:37.911670 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.911632    4471 generic.go:356] "Generic (PLEG): container finished" podID="13896eb7-f9cc-4cf2-9364-b723651037af" containerID="a6e63d232f90d0c0755e5c950af33ed9ff0a5aa077dc4d75976d435640860ab9" exitCode=0
Oct 30 21:09:37.911835 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.911685    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"
Oct 30 21:09:37.911835 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.911714    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46" event={"ID":"13896eb7-f9cc-4cf2-9364-b723651037af","Type":"ContainerDied","Data":"a6e63d232f90d0c0755e5c950af33ed9ff0a5aa077dc4d75976d435640860ab9"}
Oct 30 21:09:37.911835 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.911754    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46" event={"ID":"13896eb7-f9cc-4cf2-9364-b723651037af","Type":"ContainerDied","Data":"17dd6cc59d7dd458d783c423be43817fd1f3d634c898e0fa5d85e40fe4ebea91"}
Oct 30 21:09:37.911835 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.911770    4471 scope.go:122] "RemoveContainer" containerID="a6e63d232f90d0c0755e5c950af33ed9ff0a5aa077dc4d75976d435640860ab9"
Oct 30 21:09:37.920618 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.920598    4471 scope.go:122] "RemoveContainer" containerID="a6e63d232f90d0c0755e5c950af33ed9ff0a5aa077dc4d75976d435640860ab9"
Oct 30 21:09:37.920883 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:09:37.920863    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e63d232f90d0c0755e5c950af33ed9ff0a5aa077dc4d75976d435640860ab9\": container with ID starting with a6e63d232f90d0c0755e5c950af33ed9ff0a5aa077dc4d75976d435640860ab9 not found: ID does not exist" containerID="a6e63d232f90d0c0755e5c950af33ed9ff0a5aa077dc4d75976d435640860ab9"
Oct 30 21:09:37.920946 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:37.920892    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e63d232f90d0c0755e5c950af33ed9ff0a5aa077dc4d75976d435640860ab9"} err="failed to get container status \"a6e63d232f90d0c0755e5c950af33ed9ff0a5aa077dc4d75976d435640860ab9\": rpc error: code = NotFound desc = could not find container \"a6e63d232f90d0c0755e5c950af33ed9ff0a5aa077dc4d75976d435640860ab9\": container with ID starting with a6e63d232f90d0c0755e5c950af33ed9ff0a5aa077dc4d75976d435640860ab9 not found: ID does not exist"
Oct 30 21:09:38.075456 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:38.075428    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"]
Oct 30 21:09:38.124301 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:38.124277    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-4ee72d/simulator-w6-4ee72d-kserve-5f7d99b6db-77h46"]
Oct 30 21:09:38.429999 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:38.429966    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13896eb7-f9cc-4cf2-9364-b723651037af" path="/var/lib/kubelet/pods/13896eb7-f9cc-4cf2-9364-b723651037af/volumes"
Oct 30 21:09:48.418074 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:48.418041    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"]
Oct 30 21:09:48.418505 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:48.418236    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696" podUID="2d23e8a7-5cce-4e8b-9447-887e59719353" containerName="maas-api" containerID="cri-o://268a3b3adacb904d0f2915ea7b64eebe13ca7a9eeee298d829b9a9f33f120c95" gracePeriod=30
Oct 30 21:09:48.668854 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:48.668795    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"
Oct 30 21:09:48.687130 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:48.687101    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d23e8a7-5cce-4e8b-9447-887e59719353-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d23e8a7-5cce-4e8b-9447-887e59719353-maas-api-tls\") pod \"2d23e8a7-5cce-4e8b-9447-887e59719353\" (UID: \"2d23e8a7-5cce-4e8b-9447-887e59719353\") "
Oct 30 21:09:48.687259 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:48.687239    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d23e8a7-5cce-4e8b-9447-887e59719353-kube-api-access-rbl8k\" (UniqueName: \"kubernetes.io/projected/2d23e8a7-5cce-4e8b-9447-887e59719353-kube-api-access-rbl8k\") pod \"2d23e8a7-5cce-4e8b-9447-887e59719353\" (UID: \"2d23e8a7-5cce-4e8b-9447-887e59719353\") "
Oct 30 21:09:48.687311 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:48.687296    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d23e8a7-5cce-4e8b-9447-887e59719353-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d23e8a7-5cce-4e8b-9447-887e59719353-openshift-service-ca\") pod \"2d23e8a7-5cce-4e8b-9447-887e59719353\" (UID: \"2d23e8a7-5cce-4e8b-9447-887e59719353\") "
Oct 30 21:09:48.687372 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:48.687352    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d23e8a7-5cce-4e8b-9447-887e59719353-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d23e8a7-5cce-4e8b-9447-887e59719353-metrics-certs\") pod \"2d23e8a7-5cce-4e8b-9447-887e59719353\" (UID: \"2d23e8a7-5cce-4e8b-9447-887e59719353\") "
Oct 30 21:09:48.687766 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:48.687735    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d23e8a7-5cce-4e8b-9447-887e59719353-openshift-service-ca" pod "2d23e8a7-5cce-4e8b-9447-887e59719353" (UID: "2d23e8a7-5cce-4e8b-9447-887e59719353"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:09:48.689505 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:48.689477    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d23e8a7-5cce-4e8b-9447-887e59719353-kube-api-access-rbl8k" pod "2d23e8a7-5cce-4e8b-9447-887e59719353" (UID: "2d23e8a7-5cce-4e8b-9447-887e59719353"). InnerVolumeSpecName "kube-api-access-rbl8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:48.689611 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:48.689474    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d23e8a7-5cce-4e8b-9447-887e59719353-maas-api-tls" pod "2d23e8a7-5cce-4e8b-9447-887e59719353" (UID: "2d23e8a7-5cce-4e8b-9447-887e59719353"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:09:48.690003 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:48.689989    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d23e8a7-5cce-4e8b-9447-887e59719353-metrics-certs" pod "2d23e8a7-5cce-4e8b-9447-887e59719353" (UID: "2d23e8a7-5cce-4e8b-9447-887e59719353"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:09:48.789955 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:48.789755    4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d23e8a7-5cce-4e8b-9447-887e59719353-maas-api-tls\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:48.789955 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:48.789789    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbl8k\" (UniqueName: \"kubernetes.io/projected/2d23e8a7-5cce-4e8b-9447-887e59719353-kube-api-access-rbl8k\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:48.789955 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:48.789806    4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d23e8a7-5cce-4e8b-9447-887e59719353-openshift-service-ca\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:48.789955 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:48.789819    4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d23e8a7-5cce-4e8b-9447-887e59719353-metrics-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:09:48.964568 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:48.964472    4471 generic.go:356] "Generic (PLEG): container finished" podID="2d23e8a7-5cce-4e8b-9447-887e59719353" containerID="268a3b3adacb904d0f2915ea7b64eebe13ca7a9eeee298d829b9a9f33f120c95" exitCode=0
Oct 30 21:09:48.964568 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:48.964539    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"
Oct 30 21:09:48.964811 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:48.964566    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696" event={"ID":"2d23e8a7-5cce-4e8b-9447-887e59719353","Type":"ContainerDied","Data":"268a3b3adacb904d0f2915ea7b64eebe13ca7a9eeee298d829b9a9f33f120c95"}
Oct 30 21:09:48.964811 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:48.964631    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696" event={"ID":"2d23e8a7-5cce-4e8b-9447-887e59719353","Type":"ContainerDied","Data":"9147987df32955868237ef401ed3c89f00f9c5bba199b94d5e46ab39e73e6220"}
Oct 30 21:09:48.964811 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:48.964661    4471 scope.go:122] "RemoveContainer" containerID="268a3b3adacb904d0f2915ea7b64eebe13ca7a9eeee298d829b9a9f33f120c95"
Oct 30 21:09:48.974598 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:48.974578    4471 scope.go:122] "RemoveContainer" containerID="268a3b3adacb904d0f2915ea7b64eebe13ca7a9eeee298d829b9a9f33f120c95"
Oct 30 21:09:48.974999 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:09:48.974955    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"268a3b3adacb904d0f2915ea7b64eebe13ca7a9eeee298d829b9a9f33f120c95\": container with ID starting with 268a3b3adacb904d0f2915ea7b64eebe13ca7a9eeee298d829b9a9f33f120c95 not found: ID does not exist" containerID="268a3b3adacb904d0f2915ea7b64eebe13ca7a9eeee298d829b9a9f33f120c95"
Oct 30 21:09:48.975125 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:48.974995    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"268a3b3adacb904d0f2915ea7b64eebe13ca7a9eeee298d829b9a9f33f120c95"} err="failed to get container status \"268a3b3adacb904d0f2915ea7b64eebe13ca7a9eeee298d829b9a9f33f120c95\": rpc error: code = NotFound desc = could not find container \"268a3b3adacb904d0f2915ea7b64eebe13ca7a9eeee298d829b9a9f33f120c95\": container with ID starting with 268a3b3adacb904d0f2915ea7b64eebe13ca7a9eeee298d829b9a9f33f120c95 not found: ID does not exist"
Oct 30 21:09:49.127892 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:49.127858    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"]
Oct 30 21:09:49.132966 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:49.132940    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-4ee72d-5d87474cfd-46696"]
Oct 30 21:09:50.429822 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:50.429790    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d23e8a7-5cce-4e8b-9447-887e59719353" path="/var/lib/kubelet/pods/2d23e8a7-5cce-4e8b-9447-887e59719353/volumes"
Oct 30 21:09:57.961191 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:57.961105    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"]
Oct 30 21:09:57.961730 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:09:57.961333    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw" podUID="f068d45a-f572-4549-867a-7073169b5f05" containerName="istio-proxy" containerID="cri-o://52ca8029d60fd80360751f63d7f752f43f809a18d9f111f14db9df8c27abbacf" gracePeriod=30
Oct 30 21:10:00.725744 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:00.725708    4471 patch_prober.go:29] interesting pod/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:10:00.726130 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:00.725765    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw" podUID="f068d45a-f572-4549-867a-7073169b5f05" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:10:00.953530 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:10:00.953496    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:10:03.228156 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.228092    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 21:10:03.322297 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.322271    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-istio-envoy\") pod \"f068d45a-f572-4549-867a-7073169b5f05\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") "
Oct 30 21:10:03.322436 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.322317    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f068d45a-f572-4549-867a-7073169b5f05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f068d45a-f572-4549-867a-7073169b5f05-istiod-ca-cert\") pod \"f068d45a-f572-4549-867a-7073169b5f05\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") "
Oct 30 21:10:03.322436 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.322393    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-credential-socket\") pod \"f068d45a-f572-4549-867a-7073169b5f05\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") "
Oct 30 21:10:03.322436 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.322418    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f068d45a-f572-4549-867a-7073169b5f05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f068d45a-f572-4549-867a-7073169b5f05-istio-podinfo\") pod \"f068d45a-f572-4549-867a-7073169b5f05\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") "
Oct 30 21:10:03.322581 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.322482    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-workload-certs\") pod \"f068d45a-f572-4549-867a-7073169b5f05\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") "
Oct 30 21:10:03.322581 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.322517    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f068d45a-f572-4549-867a-7073169b5f05-kube-api-access-rxpps\" (UniqueName: \"kubernetes.io/projected/f068d45a-f572-4549-867a-7073169b5f05-kube-api-access-rxpps\") pod \"f068d45a-f572-4549-867a-7073169b5f05\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") "
Oct 30 21:10:03.322581 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.322555    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-workload-socket\") pod \"f068d45a-f572-4549-867a-7073169b5f05\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") "
Oct 30 21:10:03.322737 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.322596    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-istio-data\") pod \"f068d45a-f572-4549-867a-7073169b5f05\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") "
Oct 30 21:10:03.322737 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.322632    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f068d45a-f572-4549-867a-7073169b5f05-istio-token\" (UniqueName: \"kubernetes.io/projected/f068d45a-f572-4549-867a-7073169b5f05-istio-token\") pod \"f068d45a-f572-4549-867a-7073169b5f05\" (UID: \"f068d45a-f572-4549-867a-7073169b5f05\") "
Oct 30 21:10:03.322854 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.322730    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-credential-socket" pod "f068d45a-f572-4549-867a-7073169b5f05" (UID: "f068d45a-f572-4549-867a-7073169b5f05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:10:03.322854 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.322793    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f068d45a-f572-4549-867a-7073169b5f05-istiod-ca-cert" pod "f068d45a-f572-4549-867a-7073169b5f05" (UID: "f068d45a-f572-4549-867a-7073169b5f05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:03.322854 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.322834    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-workload-certs" pod "f068d45a-f572-4549-867a-7073169b5f05" (UID: "f068d45a-f572-4549-867a-7073169b5f05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:10:03.323065 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.323042    4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f068d45a-f572-4549-867a-7073169b5f05-istiod-ca-cert\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:03.323126 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.323073    4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-credential-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:03.323126 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.323090    4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-workload-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:03.323126 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.323101    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-istio-data" pod "f068d45a-f572-4549-867a-7073169b5f05" (UID: "f068d45a-f572-4549-867a-7073169b5f05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:10:03.323262 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.323131    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-workload-socket" pod "f068d45a-f572-4549-867a-7073169b5f05" (UID: "f068d45a-f572-4549-867a-7073169b5f05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:10:03.324877 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.324856    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f068d45a-f572-4549-867a-7073169b5f05-istio-token" pod "f068d45a-f572-4549-867a-7073169b5f05" (UID: "f068d45a-f572-4549-867a-7073169b5f05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:10:03.324989 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.324968    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f068d45a-f572-4549-867a-7073169b5f05-kube-api-access-rxpps" pod "f068d45a-f572-4549-867a-7073169b5f05" (UID: "f068d45a-f572-4549-867a-7073169b5f05"). InnerVolumeSpecName "kube-api-access-rxpps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:10:03.325224 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.325205    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f068d45a-f572-4549-867a-7073169b5f05-istio-podinfo" pod "f068d45a-f572-4549-867a-7073169b5f05" (UID: "f068d45a-f572-4549-867a-7073169b5f05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:10:03.325290 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.325217    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-istio-envoy" pod "f068d45a-f572-4549-867a-7073169b5f05" (UID: "f068d45a-f572-4549-867a-7073169b5f05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:10:03.334025 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.334004    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-756d9799fc-v9l8r"]
Oct 30 21:10:03.334253 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.334231    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-756d9799fc-v9l8r" podUID="f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b" containerName="manager" containerID="cri-o://b0344f9085fdcd49fcc5e58d2fdd812f86658b5be6fe62ee7b139a85bbf3b663" gracePeriod=30
Oct 30 21:10:03.424290 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.424264    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxpps\" (UniqueName: \"kubernetes.io/projected/f068d45a-f572-4549-867a-7073169b5f05-kube-api-access-rxpps\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:03.424290 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.424292    4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-workload-socket\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:03.424446 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.424301    4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-istio-data\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:03.424446 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.424310    4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f068d45a-f572-4549-867a-7073169b5f05-istio-token\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:03.424446 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.424333    4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f068d45a-f572-4549-867a-7073169b5f05-istio-envoy\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:03.424446 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.424343    4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f068d45a-f572-4549-867a-7073169b5f05-istio-podinfo\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:03.572714 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.572685    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-756d9799fc-v9l8r"
Oct 30 21:10:03.626976 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.626949    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b-metrics-certs\") pod \"f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b\" (UID: \"f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b\") "
Oct 30 21:10:03.627136 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.626999    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b-kube-api-access-tnw9j\" (UniqueName: \"kubernetes.io/projected/f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b-kube-api-access-tnw9j\") pod \"f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b\" (UID: \"f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b\") "
Oct 30 21:10:03.627136 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.627044    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b-webhook-cert\") pod \"f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b\" (UID: \"f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b\") "
Oct 30 21:10:03.629175 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.629143    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b-webhook-cert" pod "f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b" (UID: "f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:03.629175 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.629161    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b-metrics-certs" pod "f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b" (UID: "f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:03.629368 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.629207    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b-kube-api-access-tnw9j" pod "f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b" (UID: "f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b"). InnerVolumeSpecName "kube-api-access-tnw9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:10:03.728302 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.728264    4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b-metrics-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:03.728302 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.728303    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnw9j\" (UniqueName: \"kubernetes.io/projected/f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b-kube-api-access-tnw9j\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:03.728516 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:03.728339    4471 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b-webhook-cert\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:10:04.029561 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.029529    4471 generic.go:356] "Generic (PLEG): container finished" podID="f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b" containerID="b0344f9085fdcd49fcc5e58d2fdd812f86658b5be6fe62ee7b139a85bbf3b663" exitCode=0
Oct 30 21:10:04.029725 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.029579    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-756d9799fc-v9l8r"
Oct 30 21:10:04.029725 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.029617    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-756d9799fc-v9l8r" event={"ID":"f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b","Type":"ContainerDied","Data":"b0344f9085fdcd49fcc5e58d2fdd812f86658b5be6fe62ee7b139a85bbf3b663"}
Oct 30 21:10:04.029725 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.029676    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-756d9799fc-v9l8r" event={"ID":"f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b","Type":"ContainerDied","Data":"cebdba7aefcd5d15e15ee840bf28094a674e73bf718d51e8b370b291fe8aaa54"}
Oct 30 21:10:04.029725 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.029700    4471 scope.go:122] "RemoveContainer" containerID="b0344f9085fdcd49fcc5e58d2fdd812f86658b5be6fe62ee7b139a85bbf3b663"
Oct 30 21:10:04.031091 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.031069    4471 generic.go:356] "Generic (PLEG): container finished" podID="f068d45a-f572-4549-867a-7073169b5f05" containerID="52ca8029d60fd80360751f63d7f752f43f809a18d9f111f14db9df8c27abbacf" exitCode=0
Oct 30 21:10:04.031238 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.031103    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw" event={"ID":"f068d45a-f572-4549-867a-7073169b5f05","Type":"ContainerDied","Data":"52ca8029d60fd80360751f63d7f752f43f809a18d9f111f14db9df8c27abbacf"}
Oct 30 21:10:04.031238 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.031133    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw" event={"ID":"f068d45a-f572-4549-867a-7073169b5f05","Type":"ContainerDied","Data":"b760b8a2f9bf2434d2b93661b1bd807e1f407d07f85e9fb1b1ebed2e73a19aa1"}
Oct 30 21:10:04.031238 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.031140    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"
Oct 30 21:10:04.038715 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.038698    4471 scope.go:122] "RemoveContainer" containerID="db10ae40390eefe009b5eee2236d8cf2dc3967cf43f26a8b9c602615abd35452"
Oct 30 21:10:04.046922 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.046904    4471 scope.go:122] "RemoveContainer" containerID="b0344f9085fdcd49fcc5e58d2fdd812f86658b5be6fe62ee7b139a85bbf3b663"
Oct 30 21:10:04.047175 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:10:04.047153    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0344f9085fdcd49fcc5e58d2fdd812f86658b5be6fe62ee7b139a85bbf3b663\": container with ID starting with b0344f9085fdcd49fcc5e58d2fdd812f86658b5be6fe62ee7b139a85bbf3b663 not found: ID does not exist" containerID="b0344f9085fdcd49fcc5e58d2fdd812f86658b5be6fe62ee7b139a85bbf3b663"
Oct 30 21:10:04.047221 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.047188    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0344f9085fdcd49fcc5e58d2fdd812f86658b5be6fe62ee7b139a85bbf3b663"} err="failed to get container status \"b0344f9085fdcd49fcc5e58d2fdd812f86658b5be6fe62ee7b139a85bbf3b663\": rpc error: code = NotFound desc = could not find container \"b0344f9085fdcd49fcc5e58d2fdd812f86658b5be6fe62ee7b139a85bbf3b663\": container with ID starting with b0344f9085fdcd49fcc5e58d2fdd812f86658b5be6fe62ee7b139a85bbf3b663 not found: ID does not exist"
Oct 30 21:10:04.047221 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.047219    4471 scope.go:122] "RemoveContainer" containerID="db10ae40390eefe009b5eee2236d8cf2dc3967cf43f26a8b9c602615abd35452"
Oct 30 21:10:04.047484 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:10:04.047468    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db10ae40390eefe009b5eee2236d8cf2dc3967cf43f26a8b9c602615abd35452\": container with ID starting with db10ae40390eefe009b5eee2236d8cf2dc3967cf43f26a8b9c602615abd35452 not found: ID does not exist" containerID="db10ae40390eefe009b5eee2236d8cf2dc3967cf43f26a8b9c602615abd35452"
Oct 30 21:10:04.047533 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.047489    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db10ae40390eefe009b5eee2236d8cf2dc3967cf43f26a8b9c602615abd35452"} err="failed to get container status \"db10ae40390eefe009b5eee2236d8cf2dc3967cf43f26a8b9c602615abd35452\": rpc error: code = NotFound desc = could not find container \"db10ae40390eefe009b5eee2236d8cf2dc3967cf43f26a8b9c602615abd35452\": container with ID starting with db10ae40390eefe009b5eee2236d8cf2dc3967cf43f26a8b9c602615abd35452 not found: ID does not exist"
Oct 30 21:10:04.047533 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.047505    4471 scope.go:122] "RemoveContainer" containerID="52ca8029d60fd80360751f63d7f752f43f809a18d9f111f14db9df8c27abbacf"
Oct 30 21:10:04.054902 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.054887    4471 scope.go:122] "RemoveContainer" containerID="52ca8029d60fd80360751f63d7f752f43f809a18d9f111f14db9df8c27abbacf"
Oct 30 21:10:04.055148 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:10:04.055130    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52ca8029d60fd80360751f63d7f752f43f809a18d9f111f14db9df8c27abbacf\": container with ID starting with 52ca8029d60fd80360751f63d7f752f43f809a18d9f111f14db9df8c27abbacf not found: ID does not exist" containerID="52ca8029d60fd80360751f63d7f752f43f809a18d9f111f14db9df8c27abbacf"
Oct 30 21:10:04.055190 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.055154    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52ca8029d60fd80360751f63d7f752f43f809a18d9f111f14db9df8c27abbacf"} err="failed to get container status \"52ca8029d60fd80360751f63d7f752f43f809a18d9f111f14db9df8c27abbacf\": rpc error: code = NotFound desc = could not find container \"52ca8029d60fd80360751f63d7f752f43f809a18d9f111f14db9df8c27abbacf\": container with ID starting with 52ca8029d60fd80360751f63d7f752f43f809a18d9f111f14db9df8c27abbacf not found: ID does not exist"
Oct 30 21:10:04.191925 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.191893    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-756d9799fc-v9l8r"]
Oct 30 21:10:04.241599 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.241517    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-756d9799fc-v9l8r"]
Oct 30 21:10:04.306302 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.306273    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"]
Oct 30 21:10:04.310041 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.310016    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-4ee72d-openshift-default-689c7db79-b7mzw"]
Oct 30 21:10:04.430332 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.430288    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f068d45a-f572-4549-867a-7073169b5f05" path="/var/lib/kubelet/pods/f068d45a-f572-4549-867a-7073169b5f05/volumes"
Oct 30 21:10:04.430682 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:10:04.430669    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b" path="/var/lib/kubelet/pods/f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b/volumes"
Oct 30 21:10:30.953159 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:10:30.953123    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:11:00.953151 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:11:00.953115    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:11:30.954715 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:11:30.954622    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:12:00.953428 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:12:00.953384    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:12:30.952610 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:12:30.952576    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:13:00.953931 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:13:00.953848    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:13:30.953641 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:13:30.953604    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:14:00.953069 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:14:00.953029    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:14:30.952876 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:14:30.952791    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:15:00.953208 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:15:00.953166    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:15:30.953227 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:15:30.953188    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:16:00.953692 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:16:00.953612    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:16:30.954613 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:16:30.954574    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:16:36.576347 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:16:36.575472    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-k7458 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:16:36.576347 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:16:36.575536    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458" podUID="11141a83-7ed0-470a-9604-340b4d6af444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:00.953834 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:17:00.953795    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:17:31.022487 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:17:31.022409    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:18:00.953219 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:18:00.953175    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:18:30.953165 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:18:30.953126    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:19:00.954392 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:19:00.954356    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:19:30.956571 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:19:30.954447    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:20:00.953457 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:20:00.953421    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:20:30.953721 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:20:30.953629    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:21:00.953232 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:21:00.953194    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:21:30.982622 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:21:30.982577    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:22:00.953841 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:22:00.953764    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:22:30.953474 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:22:30.953428    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:23:00.953229 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:23:00.953187    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:23:30.952918 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:23:30.952828    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:24:00.953665 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:24:00.953625    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:24:30.953024 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:24:30.952986    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:25:00.955302 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:25:00.955220    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:25:09.317586 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:25:09.317536    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7g5ll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:25:09.318176 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:25:09.317623    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll" podUID="1ac0a5d8-9d3d-4a31-ac67-16c610d242ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:30.953234 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:25:30.953200    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:25:39.321479 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:25:39.321437    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7g5ll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:25:39.322108 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:25:39.321513    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7g5ll" podUID="1ac0a5d8-9d3d-4a31-ac67-16c610d242ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:55.478679 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:25:55.478573    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-695mh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:25:55.479402 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:25:55.479346    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-695mh" podUID="fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:00.953467 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:26:00.953431    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:26:21.582347 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:21.578434    4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-k7458 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:21.582347 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:21.578523    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7458" podUID="11141a83-7ed0-470a-9604-340b4d6af444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:30.952872 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:26:30.952783    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:26:52.624375 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624340    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-d979b6f8b-gqkwp"]
Oct 30 21:26:52.624863 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624781    4471 cpu_manager.go:414] "containerMap: removing container" podUID="811787ae-bcf5-452e-9c13-2ada1d001705" containerName="maas-api"
Oct 30 21:26:52.624863 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624797    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="811787ae-bcf5-452e-9c13-2ada1d001705" containerName="maas-api"
Oct 30 21:26:52.624863 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624816    4471 cpu_manager.go:414] "containerMap: removing container" podUID="c5652acc-b0ac-4755-a655-407ae5fba9a3" containerName="maas-api"
Oct 30 21:26:52.624863 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624832    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5652acc-b0ac-4755-a655-407ae5fba9a3" containerName="maas-api"
Oct 30 21:26:52.624863 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624842    4471 cpu_manager.go:414] "containerMap: removing container" podUID="7b68692e-aa7c-4b66-a316-19e71881d64e" containerName="istio-proxy"
Oct 30 21:26:52.624863 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624849    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b68692e-aa7c-4b66-a316-19e71881d64e" containerName="istio-proxy"
Oct 30 21:26:52.624863 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624859    4471 cpu_manager.go:414] "containerMap: removing container" podUID="da8b40b6-546e-401c-bb07-22b1fcb6791f" containerName="payload-processing"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624867    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8b40b6-546e-401c-bb07-22b1fcb6791f" containerName="payload-processing"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624877    4471 cpu_manager.go:414] "containerMap: removing container" podUID="0ec28e09-b926-4e73-8f40-2f1ab3a41df8" containerName="main"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624884    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec28e09-b926-4e73-8f40-2f1ab3a41df8" containerName="main"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624894    4471 cpu_manager.go:414] "containerMap: removing container" podUID="5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" containerName="main"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624901    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" containerName="main"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624912    4471 cpu_manager.go:414] "containerMap: removing container" podUID="096c8245-e0ed-4aa9-ad32-7aca48ab6aa4" containerName="main"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624919    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096c8245-e0ed-4aa9-ad32-7aca48ab6aa4" containerName="main"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624928    4471 cpu_manager.go:414] "containerMap: removing container" podUID="675355ae-1aae-4e8e-b12f-d3727346ae51" containerName="istio-proxy"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624935    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="675355ae-1aae-4e8e-b12f-d3727346ae51" containerName="istio-proxy"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624946    4471 cpu_manager.go:414] "containerMap: removing container" podUID="aff6334a-529c-411f-ae61-db2d2834906d" containerName="payload-processing"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624953    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff6334a-529c-411f-ae61-db2d2834906d" containerName="payload-processing"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624962    4471 cpu_manager.go:414] "containerMap: removing container" podUID="0952f140-c7e9-4e47-bf2b-a43248d73b20" containerName="maas-api"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624971    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0952f140-c7e9-4e47-bf2b-a43248d73b20" containerName="maas-api"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624981    4471 cpu_manager.go:414] "containerMap: removing container" podUID="f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b" containerName="manager"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624987    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b" containerName="manager"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.624996    4471 cpu_manager.go:414] "containerMap: removing container" podUID="a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98" containerName="main"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625002    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98" containerName="main"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625012    4471 cpu_manager.go:414] "containerMap: removing container" podUID="1c96abec-44b9-45ec-b7b4-310a2fe9a59c" containerName="cleanup"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625018    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c96abec-44b9-45ec-b7b4-310a2fe9a59c" containerName="cleanup"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625028    4471 cpu_manager.go:414] "containerMap: removing container" podUID="31cc010d-9e1f-42c2-a736-0ceacbedb8c9" containerName="main"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625034    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31cc010d-9e1f-42c2-a736-0ceacbedb8c9" containerName="main"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625045    4471 cpu_manager.go:414] "containerMap: removing container" podUID="39317fa6-714f-4826-9a67-f16b11f3f7e7" containerName="payload-pre-processing"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625051    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39317fa6-714f-4826-9a67-f16b11f3f7e7" containerName="payload-pre-processing"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625061    4471 cpu_manager.go:414] "containerMap: removing container" podUID="ea3436e4-2f27-4ec7-a5ea-4475e0717a2b" containerName="cleanup"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625068    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea3436e4-2f27-4ec7-a5ea-4475e0717a2b" containerName="cleanup"
Oct 30 21:26:52.625077 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625082    4471 cpu_manager.go:414] "containerMap: removing container" podUID="f068d45a-f572-4549-867a-7073169b5f05" containerName="istio-proxy"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625089    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f068d45a-f572-4549-867a-7073169b5f05" containerName="istio-proxy"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625098    4471 cpu_manager.go:414] "containerMap: removing container" podUID="f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b" containerName="manager"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625104    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b" containerName="manager"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625124    4471 cpu_manager.go:414] "containerMap: removing container" podUID="c120141b-7364-4435-9533-046a104f988b" containerName="istio-proxy"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625131    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c120141b-7364-4435-9533-046a104f988b" containerName="istio-proxy"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625143    4471 cpu_manager.go:414] "containerMap: removing container" podUID="ccae7148-e9b3-488b-ad71-acceff9f3d9a" containerName="maas-api"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625149    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccae7148-e9b3-488b-ad71-acceff9f3d9a" containerName="maas-api"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625159    4471 cpu_manager.go:414] "containerMap: removing container" podUID="ea3436e4-2f27-4ec7-a5ea-4475e0717a2b" containerName="cleanup"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625165    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea3436e4-2f27-4ec7-a5ea-4475e0717a2b" containerName="cleanup"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625177    4471 cpu_manager.go:414] "containerMap: removing container" podUID="13896eb7-f9cc-4cf2-9364-b723651037af" containerName="main"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625183    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13896eb7-f9cc-4cf2-9364-b723651037af" containerName="main"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625192    4471 cpu_manager.go:414] "containerMap: removing container" podUID="1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf" containerName="main"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625200    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf" containerName="main"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625209    4471 cpu_manager.go:414] "containerMap: removing container" podUID="bf89b9b8-0419-48bc-b410-12807dab75db" containerName="main"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625215    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf89b9b8-0419-48bc-b410-12807dab75db" containerName="main"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625227    4471 cpu_manager.go:414] "containerMap: removing container" podUID="a4344406-c869-420f-8cf3-93c5e0549be8" containerName="main"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625235    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4344406-c869-420f-8cf3-93c5e0549be8" containerName="main"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625244    4471 cpu_manager.go:414] "containerMap: removing container" podUID="1c96abec-44b9-45ec-b7b4-310a2fe9a59c" containerName="cleanup"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625250    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c96abec-44b9-45ec-b7b4-310a2fe9a59c" containerName="cleanup"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625296    4471 cpu_manager.go:414] "containerMap: removing container" podUID="2d23e8a7-5cce-4e8b-9447-887e59719353" containerName="maas-api"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625303    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d23e8a7-5cce-4e8b-9447-887e59719353" containerName="maas-api"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625410    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8b40b6-546e-401c-bb07-22b1fcb6791f" containerName="payload-processing"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625421    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec28e09-b926-4e73-8f40-2f1ab3a41df8" containerName="main"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625429    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bf5e57f-4c83-4c4b-9771-5f78d48d53ea" containerName="main"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625435    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="096c8245-e0ed-4aa9-ad32-7aca48ab6aa4" containerName="main"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625440    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="675355ae-1aae-4e8e-b12f-d3727346ae51" containerName="istio-proxy"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625446    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff6334a-529c-411f-ae61-db2d2834906d" containerName="payload-processing"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625451    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="0952f140-c7e9-4e47-bf2b-a43248d73b20" containerName="maas-api"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625457    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b" containerName="manager"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625461    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e9e4ae-4ffa-43ee-bce5-26b21c4c3f98" containerName="main"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625466    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c96abec-44b9-45ec-b7b4-310a2fe9a59c" containerName="cleanup"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625471    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="31cc010d-9e1f-42c2-a736-0ceacbedb8c9" containerName="main"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625476    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="39317fa6-714f-4826-9a67-f16b11f3f7e7" containerName="payload-pre-processing"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625485    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea3436e4-2f27-4ec7-a5ea-4475e0717a2b" containerName="cleanup"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625492    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="f068d45a-f572-4549-867a-7073169b5f05" containerName="istio-proxy"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625499    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="c120141b-7364-4435-9533-046a104f988b" containerName="istio-proxy"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625503    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccae7148-e9b3-488b-ad71-acceff9f3d9a" containerName="maas-api"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625510    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea3436e4-2f27-4ec7-a5ea-4475e0717a2b" containerName="cleanup"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625515    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="13896eb7-f9cc-4cf2-9364-b723651037af" containerName="main"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625519    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe1e1b3-101a-4b20-a7e9-f1c287fa12cf" containerName="main"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625524    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf89b9b8-0419-48bc-b410-12807dab75db" containerName="main"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625529    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4344406-c869-420f-8cf3-93c5e0549be8" containerName="main"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625534    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c96abec-44b9-45ec-b7b4-310a2fe9a59c" containerName="cleanup"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625539    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c96abec-44b9-45ec-b7b4-310a2fe9a59c" containerName="cleanup"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625544    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d23e8a7-5cce-4e8b-9447-887e59719353" containerName="maas-api"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625549    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="811787ae-bcf5-452e-9c13-2ada1d001705" containerName="maas-api"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625555    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5652acc-b0ac-4755-a655-407ae5fba9a3" containerName="maas-api"
Oct 30 21:26:52.625845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.625559    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b68692e-aa7c-4b66-a316-19e71881d64e" containerName="istio-proxy"
Oct 30 21:26:52.628410 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.628393    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d979b6f8b-gqkwp"
Oct 30 21:26:52.732616 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.732574    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d979b6f8b-gqkwp"]
Oct 30 21:26:52.756758 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.756729    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dcf06b4-ebfd-449f-922e-0855872bd225-metrics-certs\") pod \"maas-api-d979b6f8b-gqkwp\" (UID: \"4dcf06b4-ebfd-449f-922e-0855872bd225\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-gqkwp"
Oct 30 21:26:52.756758 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.756761    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dcf06b4-ebfd-449f-922e-0855872bd225-openshift-service-ca\") pod \"maas-api-d979b6f8b-gqkwp\" (UID: \"4dcf06b4-ebfd-449f-922e-0855872bd225\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-gqkwp"
Oct 30 21:26:52.757004 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.756788    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dcf06b4-ebfd-449f-922e-0855872bd225-maas-api-tls\") pod \"maas-api-d979b6f8b-gqkwp\" (UID: \"4dcf06b4-ebfd-449f-922e-0855872bd225\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-gqkwp"
Oct 30 21:26:52.757004 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.756909    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpkw9\" (UniqueName: \"kubernetes.io/projected/4dcf06b4-ebfd-449f-922e-0855872bd225-kube-api-access-wpkw9\") pod \"maas-api-d979b6f8b-gqkwp\" (UID: \"4dcf06b4-ebfd-449f-922e-0855872bd225\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-gqkwp"
Oct 30 21:26:52.857373 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.857340    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dcf06b4-ebfd-449f-922e-0855872bd225-maas-api-tls\") pod \"maas-api-d979b6f8b-gqkwp\" (UID: \"4dcf06b4-ebfd-449f-922e-0855872bd225\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-gqkwp"
Oct 30 21:26:52.857514 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.857401    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpkw9\" (UniqueName: \"kubernetes.io/projected/4dcf06b4-ebfd-449f-922e-0855872bd225-kube-api-access-wpkw9\") pod \"maas-api-d979b6f8b-gqkwp\" (UID: \"4dcf06b4-ebfd-449f-922e-0855872bd225\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-gqkwp"
Oct 30 21:26:52.857514 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.857453    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dcf06b4-ebfd-449f-922e-0855872bd225-metrics-certs\") pod \"maas-api-d979b6f8b-gqkwp\" (UID: \"4dcf06b4-ebfd-449f-922e-0855872bd225\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-gqkwp"
Oct 30 21:26:52.857514 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.857469    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dcf06b4-ebfd-449f-922e-0855872bd225-openshift-service-ca\") pod \"maas-api-d979b6f8b-gqkwp\" (UID: \"4dcf06b4-ebfd-449f-922e-0855872bd225\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-gqkwp"
Oct 30 21:26:52.858122 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.858098    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dcf06b4-ebfd-449f-922e-0855872bd225-openshift-service-ca\") pod \"maas-api-d979b6f8b-gqkwp\" (UID: \"4dcf06b4-ebfd-449f-922e-0855872bd225\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-gqkwp"
Oct 30 21:26:52.860071 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.860042    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dcf06b4-ebfd-449f-922e-0855872bd225-maas-api-tls\") pod \"maas-api-d979b6f8b-gqkwp\" (UID: \"4dcf06b4-ebfd-449f-922e-0855872bd225\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-gqkwp"
Oct 30 21:26:52.860456 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.860035    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dcf06b4-ebfd-449f-922e-0855872bd225-metrics-certs\") pod \"maas-api-d979b6f8b-gqkwp\" (UID: \"4dcf06b4-ebfd-449f-922e-0855872bd225\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-gqkwp"
Oct 30 21:26:52.915759 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.915705    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpkw9\" (UniqueName: \"kubernetes.io/projected/4dcf06b4-ebfd-449f-922e-0855872bd225-kube-api-access-wpkw9\") pod \"maas-api-d979b6f8b-gqkwp\" (UID: \"4dcf06b4-ebfd-449f-922e-0855872bd225\") " pod="odh-ai-gateway-infra/maas-api-d979b6f8b-gqkwp"
Oct 30 21:26:52.940661 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:52.940632    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d979b6f8b-gqkwp"
Oct 30 21:26:53.310068 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:53.309986    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d979b6f8b-gqkwp"]
Oct 30 21:26:53.313556 ip-10-0-138-34 kubenswrapper[4471]: W1030 21:26:53.313528    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dcf06b4_ebfd_449f_922e_0855872bd225.slice/crio-1c9479c033716abb8e8a818445fa834265a4dd59a52b1bb149af9da59e7e5c9b WatchSource:0}: Error finding container 1c9479c033716abb8e8a818445fa834265a4dd59a52b1bb149af9da59e7e5c9b: Status 404 returned error can't find the container with id 1c9479c033716abb8e8a818445fa834265a4dd59a52b1bb149af9da59e7e5c9b
Oct 30 21:26:53.314935 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:53.314916    4471 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:26:53.388060 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:53.388026    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d979b6f8b-gqkwp" event={"ID":"4dcf06b4-ebfd-449f-922e-0855872bd225","Type":"ContainerStarted","Data":"1c9479c033716abb8e8a818445fa834265a4dd59a52b1bb149af9da59e7e5c9b"}
Oct 30 21:26:54.393972 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:54.393934    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d979b6f8b-gqkwp" event={"ID":"4dcf06b4-ebfd-449f-922e-0855872bd225","Type":"ContainerStarted","Data":"ad81363f808471863b2d89c6e16bd672bfce1ada0dfb2f974e7f164651594a1b"}
Oct 30 21:26:54.394387 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:54.394052    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-d979b6f8b-gqkwp"
Oct 30 21:26:54.523050 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:26:54.522966    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-d979b6f8b-gqkwp" podStartSLOduration=1.5908085939999999 podStartE2EDuration="2.522952584s" totalImagesPullingTime="932.14399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:54.521249477 +0000 UTC m=+4034.199878093" watchObservedRunningTime="2026-10-30 21:26:54.522952584 +0000 UTC m=+4034.201581191"
Oct 30 21:27:00.405910 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:00.405879    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-d979b6f8b-gqkwp"
Oct 30 21:27:00.571341 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:00.571299    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"]
Oct 30 21:27:00.571539 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:00.571504    4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx" podUID="07769876-3c0c-4714-b6c7-bbd195c0d0c6" containerName="maas-api" containerID="cri-o://affab50c837207af72315b304903179d47fcb2f4988845c65611f3eae3681af3" gracePeriod=30
Oct 30 21:27:00.953173 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:27:00.953138    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:27:01.110604 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:01.110573    4471 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx" podUID="07769876-3c0c-4714-b6c7-bbd195c0d0c6" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.29:8443/health\": dial tcp 10.133.0.29:8443: connect: connection refused"
Oct 30 21:27:01.839655 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:01.839631    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"
Oct 30 21:27:01.937265 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:01.937203    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07769876-3c0c-4714-b6c7-bbd195c0d0c6-kube-api-access-tk9r2\" (UniqueName: \"kubernetes.io/projected/07769876-3c0c-4714-b6c7-bbd195c0d0c6-kube-api-access-tk9r2\") pod \"07769876-3c0c-4714-b6c7-bbd195c0d0c6\" (UID: \"07769876-3c0c-4714-b6c7-bbd195c0d0c6\") "
Oct 30 21:27:01.937265 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:01.937255    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07769876-3c0c-4714-b6c7-bbd195c0d0c6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07769876-3c0c-4714-b6c7-bbd195c0d0c6-openshift-service-ca\") pod \"07769876-3c0c-4714-b6c7-bbd195c0d0c6\" (UID: \"07769876-3c0c-4714-b6c7-bbd195c0d0c6\") "
Oct 30 21:27:01.937470 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:01.937383    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07769876-3c0c-4714-b6c7-bbd195c0d0c6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/07769876-3c0c-4714-b6c7-bbd195c0d0c6-metrics-certs\") pod \"07769876-3c0c-4714-b6c7-bbd195c0d0c6\" (UID: \"07769876-3c0c-4714-b6c7-bbd195c0d0c6\") "
Oct 30 21:27:01.937470 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:01.937416    4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07769876-3c0c-4714-b6c7-bbd195c0d0c6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07769876-3c0c-4714-b6c7-bbd195c0d0c6-maas-api-tls\") pod \"07769876-3c0c-4714-b6c7-bbd195c0d0c6\" (UID: \"07769876-3c0c-4714-b6c7-bbd195c0d0c6\") "
Oct 30 21:27:01.937733 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:01.937710    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07769876-3c0c-4714-b6c7-bbd195c0d0c6-openshift-service-ca" pod "07769876-3c0c-4714-b6c7-bbd195c0d0c6" (UID: "07769876-3c0c-4714-b6c7-bbd195c0d0c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:27:01.939981 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:01.939955    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07769876-3c0c-4714-b6c7-bbd195c0d0c6-kube-api-access-tk9r2" pod "07769876-3c0c-4714-b6c7-bbd195c0d0c6" (UID: "07769876-3c0c-4714-b6c7-bbd195c0d0c6"). InnerVolumeSpecName "kube-api-access-tk9r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:27:01.940102 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:01.940026    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07769876-3c0c-4714-b6c7-bbd195c0d0c6-maas-api-tls" pod "07769876-3c0c-4714-b6c7-bbd195c0d0c6" (UID: "07769876-3c0c-4714-b6c7-bbd195c0d0c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:27:01.940102 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:01.940041    4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07769876-3c0c-4714-b6c7-bbd195c0d0c6-metrics-certs" pod "07769876-3c0c-4714-b6c7-bbd195c0d0c6" (UID: "07769876-3c0c-4714-b6c7-bbd195c0d0c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:27:02.038255 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:02.038226    4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07769876-3c0c-4714-b6c7-bbd195c0d0c6-metrics-certs\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:02.038255 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:02.038251    4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07769876-3c0c-4714-b6c7-bbd195c0d0c6-maas-api-tls\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:02.038255 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:02.038261    4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk9r2\" (UniqueName: \"kubernetes.io/projected/07769876-3c0c-4714-b6c7-bbd195c0d0c6-kube-api-access-tk9r2\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:02.038472 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:02.038270    4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07769876-3c0c-4714-b6c7-bbd195c0d0c6-openshift-service-ca\") on node \"ip-10-0-138-34.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:27:02.430404 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:02.430375    4471 generic.go:356] "Generic (PLEG): container finished" podID="07769876-3c0c-4714-b6c7-bbd195c0d0c6" containerID="affab50c837207af72315b304903179d47fcb2f4988845c65611f3eae3681af3" exitCode=0
Oct 30 21:27:02.430576 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:02.430413    4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"
Oct 30 21:27:02.430576 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:02.430460    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx" event={"ID":"07769876-3c0c-4714-b6c7-bbd195c0d0c6","Type":"ContainerDied","Data":"affab50c837207af72315b304903179d47fcb2f4988845c65611f3eae3681af3"}
Oct 30 21:27:02.430576 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:02.430503    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-94dffd7db-65cxx" event={"ID":"07769876-3c0c-4714-b6c7-bbd195c0d0c6","Type":"ContainerDied","Data":"06e139a6259a4b07fe0b4f8ddba26bc0163f331beddfce8a694c91915e386bd1"}
Oct 30 21:27:02.430576 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:02.430525    4471 scope.go:122] "RemoveContainer" containerID="affab50c837207af72315b304903179d47fcb2f4988845c65611f3eae3681af3"
Oct 30 21:27:02.439985 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:02.439964    4471 scope.go:122] "RemoveContainer" containerID="affab50c837207af72315b304903179d47fcb2f4988845c65611f3eae3681af3"
Oct 30 21:27:02.440272 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:27:02.440246    4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"affab50c837207af72315b304903179d47fcb2f4988845c65611f3eae3681af3\": container with ID starting with affab50c837207af72315b304903179d47fcb2f4988845c65611f3eae3681af3 not found: ID does not exist" containerID="affab50c837207af72315b304903179d47fcb2f4988845c65611f3eae3681af3"
Oct 30 21:27:02.440452 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:02.440286    4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"affab50c837207af72315b304903179d47fcb2f4988845c65611f3eae3681af3"} err="failed to get container status \"affab50c837207af72315b304903179d47fcb2f4988845c65611f3eae3681af3\": rpc error: code = NotFound desc = could not find container \"affab50c837207af72315b304903179d47fcb2f4988845c65611f3eae3681af3\": container with ID starting with affab50c837207af72315b304903179d47fcb2f4988845c65611f3eae3681af3 not found: ID does not exist"
Oct 30 21:27:02.593595 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:02.593562    4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"]
Oct 30 21:27:02.642979 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:02.642946    4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-94dffd7db-65cxx"]
Oct 30 21:27:04.429139 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:27:04.429109    4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07769876-3c0c-4714-b6c7-bbd195c0d0c6" path="/var/lib/kubelet/pods/07769876-3c0c-4714-b6c7-bbd195c0d0c6/volumes"
Oct 30 21:27:30.953654 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:27:30.953602    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:28:00.953495 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:28:00.953410    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:28:30.953385 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:28:30.953344    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:29:00.953908 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:29:00.953863    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:29:30.953705 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:29:30.953620    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:30:00.953872 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:30:00.953824    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:30:30.953740 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:30:30.953703    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:31:00.955778 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:31:00.955737    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:31:30.952798 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:31:30.952759    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:31:36.864758 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:31:36.864726    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xbwkj_a7f450f5-34f5-47a4-96ed-9fb59b9544fa/manager/0.log"
Oct 30 21:31:37.544292 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:31:37.544261    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-lhj95_ce0540db-8a90-40d0-9867-73d3ee877785/manager/0.log"
Oct 30 21:31:39.125202 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:31:39.125170    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-5vs92_d131bdd1-ea77-4157-9304-8ddb194110b4/proxy/0.log"
Oct 30 21:31:40.314846 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:31:40.314809    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-d979b6f8b-gqkwp_4dcf06b4-ebfd-449f-922e-0855872bd225/maas-api/0.log"
Oct 30 21:31:41.315172 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:31:41.315140    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-zf85x_3ccee963-b721-45eb-b864-6355fa869970/postgres/0.log"
Oct 30 21:31:45.757870 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:31:45.757836    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9rshc_cf792592-4698-4010-bfb6-d6efef407058/discovery/0.log"
Oct 30 21:31:46.338436 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:31:46.338408    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-557f979cb4-v95bs_f6e9bfd6-b580-4857-990f-b23782215f70/kube-auth-proxy/0.log"
Oct 30 21:31:46.569811 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:31:46.569787    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-695mh_fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb/istio-proxy/0.log"
Oct 30 21:31:46.839992 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:31:46.839961    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-7g5ll_1ac0a5d8-9d3d-4a31-ac67-16c610d242ff/istio-proxy/0.log"
Oct 30 21:31:47.915396 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:31:47.915364    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-k7458_11141a83-7ed0-470a-9604-340b4d6af444/istio-proxy/0.log"
Oct 30 21:32:00.953945 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:32:00.953906    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:32:03.379746 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:32:03.379717    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xbwkj_a7f450f5-34f5-47a4-96ed-9fb59b9544fa/manager/0.log"
Oct 30 21:32:04.025544 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:32:04.025514    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-lhj95_ce0540db-8a90-40d0-9867-73d3ee877785/manager/0.log"
Oct 30 21:32:05.324279 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:32:05.324250    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-5vs92_d131bdd1-ea77-4157-9304-8ddb194110b4/proxy/0.log"
Oct 30 21:32:06.316235 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:32:06.316203    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-d979b6f8b-gqkwp_4dcf06b4-ebfd-449f-922e-0855872bd225/maas-api/0.log"
Oct 30 21:32:07.325292 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:32:07.325257    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-zf85x_3ccee963-b721-45eb-b864-6355fa869970/postgres/0.log"
Oct 30 21:32:11.171845 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:32:11.171811    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9rshc_cf792592-4698-4010-bfb6-d6efef407058/discovery/0.log"
Oct 30 21:32:11.757414 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:32:11.757386    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-557f979cb4-v95bs_f6e9bfd6-b580-4857-990f-b23782215f70/kube-auth-proxy/0.log"
Oct 30 21:32:11.978459 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:32:11.978432    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-695mh_fb2a6fee-e200-4ebc-a47d-2a47a7c3c9cb/istio-proxy/0.log"
Oct 30 21:32:12.202929 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:32:12.202903    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-7g5ll_1ac0a5d8-9d3d-4a31-ac67-16c610d242ff/istio-proxy/0.log"
Oct 30 21:32:13.260546 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:32:13.260514    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-k7458_11141a83-7ed0-470a-9604-340b4d6af444/istio-proxy/0.log"
Oct 30 21:32:30.953640 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:32:30.953558    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:33:00.953722 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:33:00.953674    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:33:30.954198 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:33:30.954160    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:33:37.455363 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:37.455315    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jkz4p_f399aaa3-b634-40e9-9ca7-0941f0ec2925/global-pull-secret-syncer/0.log"
Oct 30 21:33:40.427705 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:40.427674    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qxwhp_1089345a-f93e-4c78-ad1c-ac38961bdf10/konnectivity-agent/0.log"
Oct 30 21:33:41.393680 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:41.393649    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-34.us-west-1.compute.internal_d10e19b03ae8d13f3ef8ccaa1c90976f/haproxy/0.log"
Oct 30 21:33:45.881710 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.881675    4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm"]
Oct 30 21:33:45.882092 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.882010    4471 cpu_manager.go:414] "containerMap: removing container" podUID="675355ae-1aae-4e8e-b12f-d3727346ae51" containerName="istio-proxy"
Oct 30 21:33:45.882092 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.882023    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="675355ae-1aae-4e8e-b12f-d3727346ae51" containerName="istio-proxy"
Oct 30 21:33:45.882092 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.882043    4471 cpu_manager.go:414] "containerMap: removing container" podUID="07769876-3c0c-4714-b6c7-bbd195c0d0c6" containerName="maas-api"
Oct 30 21:33:45.882092 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.882051    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07769876-3c0c-4714-b6c7-bbd195c0d0c6" containerName="maas-api"
Oct 30 21:33:45.882092 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.882062    4471 cpu_manager.go:414] "containerMap: removing container" podUID="ea3436e4-2f27-4ec7-a5ea-4475e0717a2b" containerName="cleanup"
Oct 30 21:33:45.882092 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.882067    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea3436e4-2f27-4ec7-a5ea-4475e0717a2b" containerName="cleanup"
Oct 30 21:33:45.882092 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.882078    4471 cpu_manager.go:414] "containerMap: removing container" podUID="1c96abec-44b9-45ec-b7b4-310a2fe9a59c" containerName="cleanup"
Oct 30 21:33:45.882092 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.882082    4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c96abec-44b9-45ec-b7b4-310a2fe9a59c" containerName="cleanup"
Oct 30 21:33:45.882365 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.882139    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="675355ae-1aae-4e8e-b12f-d3727346ae51" containerName="istio-proxy"
Oct 30 21:33:45.882365 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.882151    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0cbbd23-0b3f-43e7-8b59-84d4cf81c14b" containerName="manager"
Oct 30 21:33:45.882365 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.882158    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="07769876-3c0c-4714-b6c7-bbd195c0d0c6" containerName="maas-api"
Oct 30 21:33:45.882365 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.882164    4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea3436e4-2f27-4ec7-a5ea-4475e0717a2b" containerName="cleanup"
Oct 30 21:33:45.885034 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.885019    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm"
Oct 30 21:33:45.936967 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.936941    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hsvt5\"/\"openshift-service-ca.crt\""
Oct 30 21:33:45.936967 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.936961    4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hsvt5\"/\"kube-root-ca.crt\""
Oct 30 21:33:45.976510 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.976481    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxfkc\" (UniqueName: \"kubernetes.io/projected/e74e57e8-00b5-4495-a92d-007f838588d3-kube-api-access-cxfkc\") pod \"perf-node-gather-daemonset-2gjnm\" (UID: \"e74e57e8-00b5-4495-a92d-007f838588d3\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm"
Oct 30 21:33:45.976663 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.976632    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e74e57e8-00b5-4495-a92d-007f838588d3-sys\") pod \"perf-node-gather-daemonset-2gjnm\" (UID: \"e74e57e8-00b5-4495-a92d-007f838588d3\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm"
Oct 30 21:33:45.976739 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.976684    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e74e57e8-00b5-4495-a92d-007f838588d3-podres\") pod \"perf-node-gather-daemonset-2gjnm\" (UID: \"e74e57e8-00b5-4495-a92d-007f838588d3\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm"
Oct 30 21:33:45.976739 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.976727    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e74e57e8-00b5-4495-a92d-007f838588d3-proc\") pod \"perf-node-gather-daemonset-2gjnm\" (UID: \"e74e57e8-00b5-4495-a92d-007f838588d3\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm"
Oct 30 21:33:45.976838 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.976754    4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e74e57e8-00b5-4495-a92d-007f838588d3-lib-modules\") pod \"perf-node-gather-daemonset-2gjnm\" (UID: \"e74e57e8-00b5-4495-a92d-007f838588d3\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm"
Oct 30 21:33:45.986221 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.986201    4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hsvt5\"/\"default-dockercfg-gxfgx\""
Oct 30 21:33:45.987669 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:45.987650    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm"]
Oct 30 21:33:46.077186 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:46.077152    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxfkc\" (UniqueName: \"kubernetes.io/projected/e74e57e8-00b5-4495-a92d-007f838588d3-kube-api-access-cxfkc\") pod \"perf-node-gather-daemonset-2gjnm\" (UID: \"e74e57e8-00b5-4495-a92d-007f838588d3\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm"
Oct 30 21:33:46.077345 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:46.077212    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e74e57e8-00b5-4495-a92d-007f838588d3-sys\") pod \"perf-node-gather-daemonset-2gjnm\" (UID: \"e74e57e8-00b5-4495-a92d-007f838588d3\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm"
Oct 30 21:33:46.077345 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:46.077252    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e74e57e8-00b5-4495-a92d-007f838588d3-podres\") pod \"perf-node-gather-daemonset-2gjnm\" (UID: \"e74e57e8-00b5-4495-a92d-007f838588d3\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm"
Oct 30 21:33:46.077345 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:46.077279    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e74e57e8-00b5-4495-a92d-007f838588d3-proc\") pod \"perf-node-gather-daemonset-2gjnm\" (UID: \"e74e57e8-00b5-4495-a92d-007f838588d3\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm"
Oct 30 21:33:46.077345 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:46.077305    4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e74e57e8-00b5-4495-a92d-007f838588d3-lib-modules\") pod \"perf-node-gather-daemonset-2gjnm\" (UID: \"e74e57e8-00b5-4495-a92d-007f838588d3\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm"
Oct 30 21:33:46.077513 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:46.077390    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e74e57e8-00b5-4495-a92d-007f838588d3-proc\") pod \"perf-node-gather-daemonset-2gjnm\" (UID: \"e74e57e8-00b5-4495-a92d-007f838588d3\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm"
Oct 30 21:33:46.077513 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:46.077391    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e74e57e8-00b5-4495-a92d-007f838588d3-sys\") pod \"perf-node-gather-daemonset-2gjnm\" (UID: \"e74e57e8-00b5-4495-a92d-007f838588d3\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm"
Oct 30 21:33:46.077513 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:46.077423    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e74e57e8-00b5-4495-a92d-007f838588d3-lib-modules\") pod \"perf-node-gather-daemonset-2gjnm\" (UID: \"e74e57e8-00b5-4495-a92d-007f838588d3\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm"
Oct 30 21:33:46.077513 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:46.077391    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e74e57e8-00b5-4495-a92d-007f838588d3-podres\") pod \"perf-node-gather-daemonset-2gjnm\" (UID: \"e74e57e8-00b5-4495-a92d-007f838588d3\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm"
Oct 30 21:33:46.134443 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:46.134381    4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxfkc\" (UniqueName: \"kubernetes.io/projected/e74e57e8-00b5-4495-a92d-007f838588d3-kube-api-access-cxfkc\") pod \"perf-node-gather-daemonset-2gjnm\" (UID: \"e74e57e8-00b5-4495-a92d-007f838588d3\") " pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm"
Oct 30 21:33:46.196749 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:46.196718    4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm"
Oct 30 21:33:46.564551 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:46.564373    4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm"]
Oct 30 21:33:46.567241 ip-10-0-138-34 kubenswrapper[4471]: W1030 21:33:46.567212    4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode74e57e8_00b5_4495_a92d_007f838588d3.slice/crio-fe7a6febcd850dd8a63653377bac7d19520cb028a5b47ef2b73f5294789c2e1e WatchSource:0}: Error finding container fe7a6febcd850dd8a63653377bac7d19520cb028a5b47ef2b73f5294789c2e1e: Status 404 returned error can't find the container with id fe7a6febcd850dd8a63653377bac7d19520cb028a5b47ef2b73f5294789c2e1e
Oct 30 21:33:47.091132 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:47.091100    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm" event={"ID":"e74e57e8-00b5-4495-a92d-007f838588d3","Type":"ContainerStarted","Data":"86724ebce3de6fba04c05e505d475b243b2964a3244b8c521a94dd81d21571b7"}
Oct 30 21:33:47.091599 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:47.091150    4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm" event={"ID":"e74e57e8-00b5-4495-a92d-007f838588d3","Type":"ContainerStarted","Data":"fe7a6febcd850dd8a63653377bac7d19520cb028a5b47ef2b73f5294789c2e1e"}
Oct 30 21:33:47.091599 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:47.091214    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm"
Oct 30 21:33:47.206342 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:47.206273    4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm" podStartSLOduration=2.206257705 podStartE2EDuration="2.206257705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:47.205783657 +0000 UTC m=+4446.884412273" watchObservedRunningTime="2026-10-30 21:33:47.206257705 +0000 UTC m=+4446.884886312"
Oct 30 21:33:53.105647 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:53.105582    4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hsvt5/perf-node-gather-daemonset-2gjnm"
Oct 30 21:33:55.212118 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:55.212085    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6d486dff54-jwr4n_7aee5968-6fa2-4193-858e-3d355c07f4b7/metrics-server/0.log"
Oct 30 21:33:56.062209 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:56.062176    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-68flc_28cd454f-5dd2-4451-b955-f09a0fadd91b/node-exporter/0.log"
Oct 30 21:33:56.479160 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:56.479115    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-68flc_28cd454f-5dd2-4451-b955-f09a0fadd91b/kube-rbac-proxy/0.log"
Oct 30 21:33:56.895956 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:33:56.895933    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-68flc_28cd454f-5dd2-4451-b955-f09a0fadd91b/init-textfile/0.log"
Oct 30 21:34:00.953781 ip-10-0-138-34 kubenswrapper[4471]: E1030 21:34:00.953747    4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6pstp" containerName="extractor"
Oct 30 21:34:05.292171 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:34:05.292140    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c8fbc85f8-pdmtt_4b16608d-45cc-4a44-b345-06e3568830f3/telemeter-client/0.log"
Oct 30 21:34:05.708372 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:34:05.708341    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c8fbc85f8-pdmtt_4b16608d-45cc-4a44-b345-06e3568830f3/reload/0.log"
Oct 30 21:34:06.125976 ip-10-0-138-34 kubenswrapper[4471]: I1030 21:34:06.125945    4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7c8fbc85f8-pdmtt_4b16608d-45cc-4a44-b345-06e3568830f3/kube-rbac-proxy/0.log"